Jul 07 18:58:45.947095 ip-10-0-141-109 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 07 18:58:45.947109 ip-10-0-141-109 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 07 18:58:45.947135 ip-10-0-141-109 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 07 18:58:45.947356 ip-10-0-141-109 systemd[1]: Failed to start Kubernetes Kubelet. Jul 07 18:58:56.147726 ip-10-0-141-109 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 07 18:58:56.147743 ip-10-0-141-109 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 713ec1ee19964d0b9ecc5c7dc73083e7 -- Jul 07 19:00:29.666353 ip-10-0-141-109 systemd[1]: Starting Kubernetes Kubelet... Jul 07 19:00:30.413637 ip-10-0-141-109 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 19:00:30.413637 ip-10-0-141-109 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 19:00:30.413637 ip-10-0-141-109 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 19:00:30.413637 ip-10-0-141-109 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 19:00:30.413637 ip-10-0-141-109 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 19:00:30.413637 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.132135 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 19:00:30.413637 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135779 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:00:30.413637 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135791 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:00:30.413637 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135795 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:00:30.413637 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135799 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:00:30.413637 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135802 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:00:30.413637 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135805 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:00:30.413637 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135808 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:00:30.413637 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135811 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:00:30.215512 ip-10-0-141-109 systemd[1]: Started Kubernetes Kubelet. Jul 07 19:00:30.420893 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135815 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:00:30.420893 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135820 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:00:30.420893 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135823 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:00:30.420893 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135826 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:00:30.420893 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135829 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:00:30.420893 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135831 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:00:30.420893 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135834 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:00:30.420893 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135839 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:00:30.420893 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135843 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:00:30.420893 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135846 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:00:30.420893 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135849 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:00:30.420893 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135852 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:00:30.420893 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135855 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:00:30.420893 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135858 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:00:30.420893 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135861 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:00:30.420893 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135865 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:00:30.420893 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135868 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:00:30.420893 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135871 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:00:30.420893 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135876 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135879 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135882 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135885 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135888 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135892 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135895 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135898 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135901 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135904 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135906 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135909 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135912 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135914 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135916 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135919 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135921 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135924 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135927 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135930 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:00:30.421728 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135932 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135936 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135939 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135942 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135944 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135947 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135949 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135952 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135954 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135956 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135959 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135961 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135964 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135967 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135970 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135972 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135975 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135978 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135980 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135983 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:00:30.422457 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135985 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:00:30.423301 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135987 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:00:30.423301 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135990 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:00:30.423301 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135992 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:00:30.423301 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135996 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:00:30.423301 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.135998 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:00:30.423301 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136002 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:00:30.423301 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136005 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:00:30.423301 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136007 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:00:30.423301 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136010 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:00:30.423301 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136013 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:00:30.423301 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136015 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:00:30.423301 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136018 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:00:30.423301 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136020 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:00:30.423301 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136023 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:00:30.423301 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136025 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:00:30.423301 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136028 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:00:30.423301 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136030 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:00:30.423301 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136032 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:00:30.423301 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136036 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136038 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136057 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136060 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136062 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136065 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136067 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136070 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136072 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136075 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136077 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136080 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136083 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136086 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136838 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136845 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136848 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136851 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136854 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136858 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:00:30.424871 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136865 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:00:30.426303 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136869 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:00:30.426303 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136872 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:00:30.426303 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136874 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:00:30.426303 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136877 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:00:30.426303 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136879 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:00:30.426303 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136882 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:00:30.426303 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136884 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:00:30.426303 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136887 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:00:30.426303 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136890 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:00:30.426303 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136892 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:00:30.426303 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136894 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:00:30.426303 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136897 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:00:30.426303 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136899 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:00:30.426303 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136903 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:00:30.426303 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136905 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:00:30.426303 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136908 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:00:30.426303 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136911 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:00:30.426303 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136913 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:00:30.426303 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136917 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136921 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136925 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136928 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136931 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136934 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136937 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136939 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136942 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136944 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136947 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136949 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136952 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136957 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136960 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136962 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136965 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136968 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136970 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136973 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:00:30.442715 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136975 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136978 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136980 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136983 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136985 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136987 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136990 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136992 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136995 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.136998 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137000 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137003 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137005 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137008 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137010 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137013 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137015 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137019 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137022 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137024 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:00:30.443423 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137026 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:00:30.444153 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137030 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:00:30.444153 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137033 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:00:30.444153 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137037 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:00:30.444153 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137059 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:00:30.444153 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137064 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:00:30.444153 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137068 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:00:30.444153 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137071 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:00:30.444153 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137074 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:00:30.444153 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137077 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:00:30.444153 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137080 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:00:30.444153 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137082 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:00:30.444153 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137085 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:00:30.444153 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137088 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:00:30.444153 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137091 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:00:30.444153 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137093 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:00:30.444153 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137096 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:00:30.444153 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137098 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:00:30.444153 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137101 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:00:30.444153 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137103 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:00:30.445016 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137106 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:00:30.445016 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137110 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:00:30.445016 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137113 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:00:30.445016 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137116 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:00:30.445016 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137119 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:00:30.445016 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137122 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:00:30.445016 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137124 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:00:30.445016 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137127 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:00:30.445016 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137129 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:00:30.445016 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137131 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:00:30.445016 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137134 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:00:30.445016 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137136 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:00:30.445016 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137139 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:00:30.445016 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137141 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:00:30.445016 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137499 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:00:30.445016 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137503 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:00:30.445016 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137512 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:00:30.445016 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137516 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:00:30.445016 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137519 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137522 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137524 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137527 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137529 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137532 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137534 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137537 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137539 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137542 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137544 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137547 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137549 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137552 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137554 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137556 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137559 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137562 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137564 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137567 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:00:30.445648 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137570 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137573 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137575 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137578 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137580 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137582 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137585 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137587 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137590 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137592 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137596 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137600 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137603 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137606 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137608 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137611 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137613 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137616 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137619 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137621 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:00:30.446245 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137624 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:00:30.446850 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137626 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:00:30.446850 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137629 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:00:30.446850 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137632 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:00:30.446850 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137635 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:00:30.446850 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137638 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:00:30.446850 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137640 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:00:30.446850 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137642 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:00:30.446850 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137645 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:00:30.446850 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137650 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:00:30.446850 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137653 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:00:30.446850 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137656 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:00:30.446850 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137659 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:00:30.446850 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137663 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:00:30.446850 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137667 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:00:30.446850 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137671 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:00:30.446850 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137675 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:00:30.446850 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137679 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:00:30.446850 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137681 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:00:30.446850 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137684 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137687 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137689 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137693 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137697 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137699 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137702 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137705 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137708 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137710 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137713 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137716 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137718 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137720 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137723 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137725 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137728 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137730 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137733 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137735 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:00:30.447419 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137737 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137740 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137742 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137745 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137747 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137749 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137752 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137754 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137756 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137759 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137761 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137764 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137766 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137769 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137773 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.137777 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137855 2568 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137865 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137874 2568 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137881 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137887 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 19:00:30.448010 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137892 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137898 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137905 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137908 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137911 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137914 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137917 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137920 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137923 2568 flags.go:64] FLAG: --cgroup-root="" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137926 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137929 2568 flags.go:64] FLAG: --client-ca-file="" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137932 2568 flags.go:64] FLAG: --cloud-provider="external" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137935 2568 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137939 2568 flags.go:64] FLAG: --cluster-domain="" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137942 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137945 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137948 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137951 2568 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137955 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137957 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137960 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137964 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137967 2568 flags.go:64] FLAG: --contention-profiling="false" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137969 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 19:00:30.448573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137972 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137975 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137979 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137985 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137988 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137991 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137993 2568 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.137996 2568 flags.go:64] FLAG: --enable-server="true" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138000 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138023 2568 flags.go:64] FLAG: --event-burst="100" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138027 2568 flags.go:64] FLAG: --event-qps="50" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138030 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138034 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138037 2568 flags.go:64] FLAG: --eviction-hard="" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138059 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138062 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138066 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138069 2568 flags.go:64] FLAG: --eviction-soft="" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138072 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138075 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138078 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138081 2568 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138083 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138086 2568 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138089 2568 flags.go:64] FLAG: --feature-gates="" Jul 07 19:00:30.449248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138093 2568 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138096 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138099 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138102 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138105 2568 flags.go:64] FLAG: --healthz-port="10248" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138108 2568 flags.go:64] FLAG: --help="false" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138111 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-141-109.ec2.internal" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138114 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138117 2568 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138121 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138125 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138129 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138132 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138136 2568 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138138 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138141 2568 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138145 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138149 2568 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138151 2568 flags.go:64] FLAG: --kube-reserved="" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138154 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138157 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138160 2568 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138163 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138166 2568 flags.go:64] FLAG: --lock-file="" Jul 07 19:00:30.449904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138168 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138171 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138185 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138191 2568 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138194 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138197 2568 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138200 2568 flags.go:64] FLAG: --logging-format="text" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138203 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138206 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138209 2568 flags.go:64] FLAG: --manifest-url="" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138212 2568 flags.go:64] FLAG: --manifest-url-header="" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138216 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138219 2568 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138223 2568 flags.go:64] FLAG: --max-pods="110" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138226 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138229 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138232 2568 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138236 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138241 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138244 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138246 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138253 2568 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138256 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138259 2568 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 19:00:30.450602 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138262 2568 flags.go:64] FLAG: --pod-cidr="" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138266 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138271 2568 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138274 2568 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138278 2568 flags.go:64] FLAG: --pods-per-core="0" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138280 2568 flags.go:64] FLAG: --port="10250" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138283 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138286 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03dc13a7f8a0ef898" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138290 2568 flags.go:64] FLAG: --qos-reserved="" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138292 2568 flags.go:64] FLAG: --read-only-port="10255" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138296 2568 flags.go:64] FLAG: --register-node="true" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138298 2568 flags.go:64] FLAG: --register-with-taints="" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138302 2568 flags.go:64] FLAG: --registry-burst="10" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138304 2568 flags.go:64] FLAG: --registry-qps="5" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138307 2568 flags.go:64] FLAG: --reserved-cpus="" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138310 2568 flags.go:64] FLAG: --reserved-memory="" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138314 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138317 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138319 2568 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138322 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138325 2568 flags.go:64] FLAG: --runonce="false" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138328 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138331 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138334 2568 flags.go:64] FLAG: --seccomp-default="false" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138337 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 19:00:30.451682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138341 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138344 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138348 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138351 2568 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138354 2568 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138357 2568 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138360 2568 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138363 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138366 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138369 2568 flags.go:64] FLAG: --system-cgroups="" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138372 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138378 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138381 2568 flags.go:64] FLAG: --tls-cert-file="" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138383 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138387 2568 flags.go:64] FLAG: --tls-min-version="" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138390 2568 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138393 2568 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138396 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138399 2568 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138402 2568 flags.go:64] FLAG: --v="2" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138405 2568 flags.go:64] FLAG: --version="false" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138410 2568 flags.go:64] FLAG: --vmodule="" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138413 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138416 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138530 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:00:30.452430 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138534 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138537 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138540 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138543 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138545 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138548 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138551 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138564 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138566 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138571 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138573 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138576 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138578 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138581 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138584 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138586 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138589 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138593 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138595 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138598 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:00:30.563977 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138600 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:00:30.564821 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138603 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:00:30.564821 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138605 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:00:30.564821 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138608 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:00:30.564821 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138610 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:00:30.564821 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138613 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:00:30.564821 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138615 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:00:30.564821 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138618 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:00:30.564821 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138620 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:00:30.564821 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138623 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:00:30.564821 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138625 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:00:30.564821 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138628 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:00:30.564821 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138630 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:00:30.564821 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138633 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:00:30.564821 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138637 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:00:30.564821 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138641 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:00:30.564821 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138645 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:00:30.564821 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138648 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:00:30.564821 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138651 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:00:30.564821 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138655 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:00:30.565363 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138658 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:00:30.565363 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138662 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:00:30.565363 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138664 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:00:30.565363 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138667 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:00:30.565363 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138670 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:00:30.565363 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138672 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:00:30.565363 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138675 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:00:30.565363 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138677 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:00:30.565363 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138680 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:00:30.565363 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138682 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:00:30.565363 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138685 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:00:30.565363 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138688 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:00:30.565363 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138690 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:00:30.565363 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138694 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:00:30.565363 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138696 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:00:30.565363 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138699 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:00:30.565363 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138701 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:00:30.565363 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138705 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:00:30.565363 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138708 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138711 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138714 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138716 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138718 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138721 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138724 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138726 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138729 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138731 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138734 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138736 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138739 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138744 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138746 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138750 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138752 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138755 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138757 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138760 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:00:30.565906 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138763 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:00:30.566505 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138765 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:00:30.566505 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138767 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:00:30.566505 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138770 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:00:30.566505 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138772 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:00:30.566505 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138775 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:00:30.566505 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138778 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:00:30.566505 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138780 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:00:30.566505 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138783 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:00:30.566505 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138785 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:00:30.566505 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138788 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:00:30.566505 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138790 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:00:30.566505 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138793 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:00:30.566505 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138795 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:00:30.566505 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138797 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:00:30.566505 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138800 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:00:30.566505 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138802 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:00:30.566505 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138805 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:00:30.566505 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138807 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:00:30.566505 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.138810 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:00:30.567083 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.138815 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 19:00:30.567083 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.145763 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 19:00:30.567083 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.145777 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 19:00:30.567083 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.145798 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 19:00:30.567083 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.145804 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 19:00:30.567083 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145861 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:00:30.567083 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145866 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:00:30.567083 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145869 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:00:30.567083 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145872 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:00:30.567083 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145875 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:00:30.567083 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145878 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:00:30.567083 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145881 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:00:30.567083 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145884 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:00:30.567083 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145887 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:00:30.567083 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145891 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:00:30.567503 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145895 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:00:30.567503 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145898 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:00:30.567503 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145901 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:00:30.567503 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145904 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:00:30.567503 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145907 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:00:30.567503 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145910 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:00:30.567503 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145915 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:00:30.567503 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145918 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:00:30.567503 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145921 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:00:30.567503 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145924 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:00:30.567503 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145927 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:00:30.567503 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145930 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:00:30.567503 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145932 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:00:30.567503 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145935 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:00:30.567503 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145938 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:00:30.567503 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145940 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:00:30.567503 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145944 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:00:30.567503 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145948 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:00:30.567503 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145951 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145955 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145957 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145960 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145963 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145966 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145970 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145973 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145976 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145978 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145981 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145983 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145986 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145989 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145991 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145993 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145996 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.145999 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146002 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146005 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:00:30.568067 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146009 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146012 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146015 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146017 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146020 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146022 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146025 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146028 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146030 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146033 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146036 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146038 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146058 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146063 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146067 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146072 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146075 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146077 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146081 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146083 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:00:30.568609 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146086 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:00:30.569200 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146089 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:00:30.569200 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146092 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:00:30.569200 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146094 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:00:30.569200 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146097 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:00:30.569200 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146099 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:00:30.569200 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146102 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:00:30.569200 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146104 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:00:30.569200 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146107 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:00:30.569200 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146110 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:00:30.569200 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146114 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:00:30.569200 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146116 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:00:30.569200 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146120 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:00:30.569200 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146123 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:00:30.569200 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146126 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:00:30.569200 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146128 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:00:30.569200 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146131 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:00:30.569200 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146133 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:00:30.569200 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146135 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:00:30.569200 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146138 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:00:30.569707 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146140 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:00:30.569707 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146143 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:00:30.569707 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146145 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:00:30.569707 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146148 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:00:30.569707 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146151 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:00:30.569707 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146153 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:00:30.569707 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146156 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:00:30.569707 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146159 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:00:30.569707 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146161 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:00:30.569707 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146164 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:00:30.569707 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146167 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:00:30.569707 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.146172 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 19:00:30.569707 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146286 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:00:30.569707 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146291 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:00:30.569707 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146294 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146297 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146300 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146303 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146306 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146308 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146311 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146314 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146316 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146320 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146324 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146326 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146329 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146333 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146337 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146340 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146343 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146346 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146348 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146351 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:00:30.570202 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146354 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:00:30.570778 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146356 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:00:30.570778 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146360 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:00:30.570778 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146363 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:00:30.570778 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146366 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:00:30.570778 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146369 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:00:30.570778 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146372 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:00:30.570778 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146375 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:00:30.570778 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146378 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:00:30.570778 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146381 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:00:30.570778 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146384 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:00:30.570778 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146386 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:00:30.570778 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146389 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:00:30.570778 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146391 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:00:30.570778 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146394 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:00:30.570778 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146396 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:00:30.570778 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146399 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:00:30.570778 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146402 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:00:30.570778 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146404 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:00:30.570778 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146407 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146409 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146413 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146417 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146420 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146423 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146425 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146427 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146430 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146432 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146435 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146438 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146440 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146442 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146445 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146448 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146450 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146453 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146455 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146457 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:00:30.571326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146460 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146463 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146465 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146469 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146472 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146475 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146477 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146480 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146482 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146485 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146487 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146490 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146492 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146495 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146499 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146503 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146506 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146509 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146511 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146514 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:00:30.571863 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146516 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:00:30.572442 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146519 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:00:30.572442 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146521 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:00:30.572442 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146524 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:00:30.572442 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146526 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:00:30.572442 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146529 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:00:30.572442 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146531 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:00:30.572442 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146534 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:00:30.572442 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146536 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:00:30.572442 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146539 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:00:30.572442 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146541 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:00:30.572442 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146544 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:00:30.572442 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146546 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:00:30.572442 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146549 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:00:30.572442 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146551 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:00:30.572442 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146563 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:00:30.572442 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146568 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:00:30.572442 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:30.146570 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:00:30.572886 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.146575 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 19:00:30.572886 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.146684 2568 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 19:00:30.572886 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.148932 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 19:00:30.572886 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.150818 2568 server.go:1019] "Starting client certificate rotation" Jul 07 19:00:30.572886 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.150917 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 19:00:30.572886 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.150952 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 19:00:30.572886 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.156570 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 19:00:30.572886 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.158670 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 19:00:30.572886 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.176559 2568 log.go:25] "Validated CRI v1 runtime API" Jul 07 19:00:30.572886 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.179774 2568 log.go:25] "Validated CRI v1 image API" Jul 07 19:00:30.572886 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.180560 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 19:00:30.572886 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.181397 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 19:00:30.573279 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.183360 2568 fs.go:135] Filesystem UUIDs: map[00b57133-d253-4714-815d-1662d6c732a8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 bd4d579b-c126-4612-acf3-13e46fd9e363:/dev/nvme0n1p3] Jul 07 19:00:30.573279 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.183380 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 19:00:30.573454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.188780 2568 manager.go:216] Machine: {Timestamp:2026-07-07 19:00:30.18710116 +0000 UTC m=+0.379641217 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100044 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2609f8aea793d9d9b9ae1130f08f7f SystemUUID:ec2609f8-aea7-93d9-d9b9-ae1130f08f7f BootID:713ec1ee-1996-4d0b-9ecc-5c7dc73083e7 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:00:a7:aa:23:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:00:a7:aa:23:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:a8:8e:cb:c9:b4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 19:00:30.573454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.188889 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 19:00:30.573454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.188973 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 19:00:30.573454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.192867 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 19:00:30.573454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.192892 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-109.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 19:00:30.573454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.194214 2568 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 19:00:30.573454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.194224 2568 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 19:00:30.573454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.194233 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 19:00:30.573454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.195010 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 19:00:30.573454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.196465 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 19:00:30.573454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.200109 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 07 19:00:30.573454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.200232 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 19:00:30.573454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.200261 2568 kubelet.go:486] "Attempting to sync node with API server" Jul 07 19:00:30.573454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.200271 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 19:00:30.573454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.200286 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 19:00:30.573454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.200295 2568 kubelet.go:398] "Adding apiserver pod source" Jul 07 19:00:30.573454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.200304 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 19:00:30.573454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.200713 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.200728 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.205845 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.208106 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.208134 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.209312 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.209325 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.209331 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.209339 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.209348 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.209354 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.209361 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.209366 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.209373 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.209378 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.209387 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.209392 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.209802 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.209833 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-109.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.211424 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.211434 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.213822 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-109.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.214614 2568 server.go:1268] "Started kubelet" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.214693 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.214710 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.214783 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.215185 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nkkh7" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.216430 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.217003 2568 server.go:310] "Adding debug handlers to kubelet server" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.221740 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nkkh7" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.222950 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.222961 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.223037 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-109.ec2.internal\" not found" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.223626 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.223629 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.223651 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.223759 2568 reconstruct.go:97] "Volume reconstruction finished" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.223766 2568 reconciler.go:29] "Reconciler: start to sync state" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.224985 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.225219 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.225229 2568 factory.go:55] Registering systemd factory Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.225238 2568 factory.go:223] Registration of the systemd container factory successfully Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.225428 2568 factory.go:153] Registering CRI-O factory Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.225441 2568 factory.go:223] Registration of the crio container factory successfully Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.225463 2568 factory.go:103] Registering Raw factory Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.225476 2568 manager.go:1196] Started watching for new ooms in manager Jul 07 19:00:30.675512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.226184 2568 manager.go:349] Starting recovery of all containers Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.228871 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-109.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.229395 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.228908 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-109.ec2.internal.18c01714b950ea5a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-109.ec2.internal,UID:ip-10-0-141-109.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-109.ec2.internal,},FirstTimestamp:2026-07-07 19:00:30.214589018 +0000 UTC m=+0.407129076,LastTimestamp:2026-07-07 19:00:30.214589018 +0000 UTC m=+0.407129076,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-109.ec2.internal,}" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.237588 2568 manager.go:354] Recovery completed Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.243272 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.245149 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.245179 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.245189 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.245652 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.245664 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.245667 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.245679 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.246347 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.246368 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.246378 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.247069 2568 policy_none.go:49] "None policy: Start" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.247086 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.247100 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.248379 2568 policy_none.go:47] "Start" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.288708 2568 manager.go:337] "Starting Device Plugin manager" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.288770 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.288780 2568 server.go:87] "Starting device plugin registration server" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.289029 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.289062 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.289325 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.289604 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.289615 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.289731 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.289776 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-109.ec2.internal\" not found" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.320973 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.322111 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.322149 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.322184 2568 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.322221 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.324959 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.390108 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.391063 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.391096 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.391107 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:00:30.676810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.391128 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-109.ec2.internal" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.400764 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-109.ec2.internal" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.400788 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-109.ec2.internal\": node \"ip-10-0-141-109.ec2.internal\" not found" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.419846 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-109.ec2.internal\" not found" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.422512 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-109.ec2.internal"] Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.422775 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.423648 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.423680 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.423694 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.423938 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13f9b3936a5fcd68b0bd917da8308fa9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal\" (UID: \"13f9b3936a5fcd68b0bd917da8308fa9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.423958 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13f9b3936a5fcd68b0bd917da8308fa9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal\" (UID: \"13f9b3936a5fcd68b0bd917da8308fa9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.426283 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.426440 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.426467 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.427067 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.427082 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.427106 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.427107 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.427121 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.427154 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.428480 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-109.ec2.internal" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.428507 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.429670 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.429696 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.429709 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.470919 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-109.ec2.internal\" not found" node="ip-10-0-141-109.ec2.internal" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.474874 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-109.ec2.internal\" not found" node="ip-10-0-141-109.ec2.internal" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.520327 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-109.ec2.internal\" not found" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.524587 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13f9b3936a5fcd68b0bd917da8308fa9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal\" (UID: \"13f9b3936a5fcd68b0bd917da8308fa9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.524621 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13f9b3936a5fcd68b0bd917da8308fa9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal\" (UID: \"13f9b3936a5fcd68b0bd917da8308fa9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal" Jul 07 19:00:30.678655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.524927 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13f9b3936a5fcd68b0bd917da8308fa9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal\" (UID: \"13f9b3936a5fcd68b0bd917da8308fa9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal" Jul 07 19:00:30.679491 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.524659 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13f9b3936a5fcd68b0bd917da8308fa9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal\" (UID: \"13f9b3936a5fcd68b0bd917da8308fa9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal" Jul 07 19:00:30.679491 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.620423 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-109.ec2.internal\" not found" Jul 07 19:00:30.679491 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.624724 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3bad345f280d0902b6e7e2ae30ad5b4d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-109.ec2.internal\" (UID: \"3bad345f280d0902b6e7e2ae30ad5b4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-109.ec2.internal" Jul 07 19:00:30.786402 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.721122 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-109.ec2.internal\" not found" Jul 07 19:00:30.786402 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.725428 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3bad345f280d0902b6e7e2ae30ad5b4d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-109.ec2.internal\" (UID: \"3bad345f280d0902b6e7e2ae30ad5b4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-109.ec2.internal" Jul 07 19:00:30.786402 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.725476 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3bad345f280d0902b6e7e2ae30ad5b4d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-109.ec2.internal\" (UID: \"3bad345f280d0902b6e7e2ae30ad5b4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-109.ec2.internal" Jul 07 19:00:30.786402 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.774576 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal" Jul 07 19:00:30.786402 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:30.778993 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-109.ec2.internal" Jul 07 19:00:30.921573 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.822101 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-109.ec2.internal\" not found" Jul 07 19:00:30.922999 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:30.922630 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-109.ec2.internal\" not found" Jul 07 19:00:31.050854 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:31.023134 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-109.ec2.internal\" not found" Jul 07 19:00:31.180155 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:31.123182 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-109.ec2.internal\" not found" Jul 07 19:00:31.180155 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.150276 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 19:00:31.180155 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.150445 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.194538 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.200669 2568 apiserver.go:52] "Watching apiserver" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.211018 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.213280 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-drmdz","openshift-multus/network-metrics-daemon-rfz5t","openshift-network-operator/iptables-alerter-45jxl","kube-system/global-pull-secret-syncer-n6cxp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd","openshift-cluster-node-tuning-operator/tuned-ldtzz","openshift-image-registry/node-ca-vrwmc","openshift-multus/multus-additional-cni-plugins-g2vp4","openshift-network-diagnostics/network-check-target-8m6sh","openshift-ovn-kubernetes/ovnkube-node-sstqk","kube-system/konnectivity-agent-hvj8p","openshift-dns/node-resolver-ptk5w"] Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.214961 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.216477 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.217251 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pqr72\"" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.217261 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.217267 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.217320 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.217478 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.218280 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.218404 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ptk5w" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.218452 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.218588 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6v5zg\"" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.218753 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.220026 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:31.220122 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfz5t" podUID="848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.220784 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-95q2m\"" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.220831 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.220915 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.221765 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-45jxl" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.223239 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.223263 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.223900 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:31.223964 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n6cxp" podUID="c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.223972 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.227742 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.228201 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 18:55:30 +0000 UTC" deadline="2028-04-18 06:35:24.40584978 +0000 UTC" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.228248 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15611h34m53.177606214s" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.228263 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xzqls\"" Jul 07 19:00:31.282841 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.228280 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 19:00:31.284031 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.228996 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-dbus\") pod \"global-pull-secret-syncer-n6cxp\" (UID: \"c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac\") " pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:31.284031 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229061 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-kubernetes\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.284031 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229098 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-systemd\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.284031 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229141 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-tuned\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.284031 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229174 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-etc-kubernetes\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.284031 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229206 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f935f581-e0c8-48ec-aebb-0fd2444902d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.284031 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229241 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-system-cni-dir\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.284031 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229272 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-cnibin\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.284031 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229331 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-cni-binary-copy\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.284031 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229386 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-host-run-k8s-cni-cncf-io\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.284031 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229436 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-host-var-lib-cni-bin\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.284031 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229470 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4bc4\" (UniqueName: \"kubernetes.io/projected/f935f581-e0c8-48ec-aebb-0fd2444902d9-kube-api-access-h4bc4\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.284031 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229500 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a079670f-9732-4279-901b-5fc3310a9a84-hosts-file\") pod \"node-resolver-ptk5w\" (UID: \"a079670f-9732-4279-901b-5fc3310a9a84\") " pod="openshift-dns/node-resolver-ptk5w" Jul 07 19:00:31.284031 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229525 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a079670f-9732-4279-901b-5fc3310a9a84-tmp-dir\") pod \"node-resolver-ptk5w\" (UID: \"a079670f-9732-4279-901b-5fc3310a9a84\") " pod="openshift-dns/node-resolver-ptk5w" Jul 07 19:00:31.284031 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229556 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-kubelet-config\") pod \"global-pull-secret-syncer-n6cxp\" (UID: \"c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac\") " pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:31.284031 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229588 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret\") pod \"global-pull-secret-syncer-n6cxp\" (UID: \"c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac\") " pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:31.284031 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229639 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs\") pod \"network-metrics-daemon-rfz5t\" (UID: \"848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c\") " pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:31.284858 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229675 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-host-var-lib-cni-multus\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.284858 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229707 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crxh5\" (UniqueName: \"kubernetes.io/projected/db094710-2848-4e1b-a04f-bc0733b68e4e-kube-api-access-crxh5\") pod \"iptables-alerter-45jxl\" (UID: \"db094710-2848-4e1b-a04f-bc0733b68e4e\") " pod="openshift-network-operator/iptables-alerter-45jxl" Jul 07 19:00:31.284858 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229735 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-modprobe-d\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.284858 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229785 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-sysconfig\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.284858 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229819 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-sysctl-conf\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.284858 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229851 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-lib-modules\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.284858 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229882 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-var-lib-kubelet\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.284858 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229908 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-hostroot\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.284858 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.229945 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwxrn\" (UniqueName: \"kubernetes.io/projected/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-kube-api-access-zwxrn\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.284858 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.230044 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f935f581-e0c8-48ec-aebb-0fd2444902d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.284858 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.230144 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f935f581-e0c8-48ec-aebb-0fd2444902d9-socket-dir\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.284858 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.230340 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f935f581-e0c8-48ec-aebb-0fd2444902d9-registration-dir\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.284858 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.230375 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f935f581-e0c8-48ec-aebb-0fd2444902d9-device-dir\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.284858 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.230402 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n55d\" (UniqueName: \"kubernetes.io/projected/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-kube-api-access-4n55d\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.284858 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.230433 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-host-var-lib-kubelet\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.284858 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.230464 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-multus-conf-dir\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.285547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.230495 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-multus-daemon-config\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.285547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.230577 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lwg9\" (UniqueName: \"kubernetes.io/projected/a079670f-9732-4279-901b-5fc3310a9a84-kube-api-access-2lwg9\") pod \"node-resolver-ptk5w\" (UID: \"a079670f-9732-4279-901b-5fc3310a9a84\") " pod="openshift-dns/node-resolver-ptk5w" Jul 07 19:00:31.285547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.230659 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db094710-2848-4e1b-a04f-bc0733b68e4e-host-slash\") pod \"iptables-alerter-45jxl\" (UID: \"db094710-2848-4e1b-a04f-bc0733b68e4e\") " pod="openshift-network-operator/iptables-alerter-45jxl" Jul 07 19:00:31.285547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.230715 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-host\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.285547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.230754 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-tmp\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.285547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.230797 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-multus-cni-dir\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.285547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.230822 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-os-release\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.285547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.230901 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-multus-socket-dir-parent\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.285547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.230935 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-host-run-netns\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.285547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.230971 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-host-run-multus-certs\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.285547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.231000 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-run\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.285547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.231027 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-sys\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.285547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.231075 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f935f581-e0c8-48ec-aebb-0fd2444902d9-sys-fs\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.285547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.231114 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c7pr\" (UniqueName: \"kubernetes.io/projected/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-kube-api-access-5c7pr\") pod \"network-metrics-daemon-rfz5t\" (UID: \"848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c\") " pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:31.285547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.231166 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db094710-2848-4e1b-a04f-bc0733b68e4e-iptables-alerter-script\") pod \"iptables-alerter-45jxl\" (UID: \"db094710-2848-4e1b-a04f-bc0733b68e4e\") " pod="openshift-network-operator/iptables-alerter-45jxl" Jul 07 19:00:31.285547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.231196 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-sysctl-d\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.285547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.235680 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.285547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.237315 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vrwmc" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.237887 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.238401 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.238490 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-spjkw\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.239075 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.239387 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.239696 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.240977 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:31.241044 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m6sh" podUID="1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.241811 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-g5td5\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.241943 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.242149 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hcwcs\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.242538 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.242802 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.243223 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.243602 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.244580 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hvj8p" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.245659 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal"] Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.245786 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.245849 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.245999 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-109.ec2.internal" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.246273 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.246503 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.246773 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.246937 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mqmcg\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.247083 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.247214 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.247264 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.247383 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-nfmlp\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.247482 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.254431 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-109.ec2.internal"] Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.255444 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.272642 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-66kzk" Jul 07 19:00:31.286233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.281283 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-66kzk" Jul 07 19:00:31.381840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.324767 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 19:00:31.381840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.333715 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-kubernetes\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.381840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.333744 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n55d\" (UniqueName: \"kubernetes.io/projected/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-kube-api-access-4n55d\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.381840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.333765 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-kubelet\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.381840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.333811 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-etc-kubernetes\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.381840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.333861 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-cnibin\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.381840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.333876 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-kubernetes\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.381840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.333871 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-etc-kubernetes\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.381840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.333898 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-cni-binary-copy\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.381840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.333938 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a079670f-9732-4279-901b-5fc3310a9a84-hosts-file\") pod \"node-resolver-ptk5w\" (UID: \"a079670f-9732-4279-901b-5fc3310a9a84\") " pod="openshift-dns/node-resolver-ptk5w" Jul 07 19:00:31.381840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.333916 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-cnibin\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.381840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.334340 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a079670f-9732-4279-901b-5fc3310a9a84-hosts-file\") pod \"node-resolver-ptk5w\" (UID: \"a079670f-9732-4279-901b-5fc3310a9a84\") " pod="openshift-dns/node-resolver-ptk5w" Jul 07 19:00:31.381840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.334373 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a079670f-9732-4279-901b-5fc3310a9a84-tmp-dir\") pod \"node-resolver-ptk5w\" (UID: \"a079670f-9732-4279-901b-5fc3310a9a84\") " pod="openshift-dns/node-resolver-ptk5w" Jul 07 19:00:31.381840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.334425 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret\") pod \"global-pull-secret-syncer-n6cxp\" (UID: \"c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac\") " pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:31.381840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.334481 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-host-var-lib-cni-bin\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.381840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.334519 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-kubelet-config\") pod \"global-pull-secret-syncer-n6cxp\" (UID: \"c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac\") " pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:31.381840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.334535 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a079670f-9732-4279-901b-5fc3310a9a84-tmp-dir\") pod \"node-resolver-ptk5w\" (UID: \"a079670f-9732-4279-901b-5fc3310a9a84\") " pod="openshift-dns/node-resolver-ptk5w" Jul 07 19:00:31.381840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.334546 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs\") pod \"network-metrics-daemon-rfz5t\" (UID: \"848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c\") " pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:31.381840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.334577 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-sysconfig\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.382765 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.334683 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-host-var-lib-cni-bin\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.382765 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.334707 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-kubelet-config\") pod \"global-pull-secret-syncer-n6cxp\" (UID: \"c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac\") " pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:31.382765 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.334753 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-sysconfig\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.382765 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.334797 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-var-lib-kubelet\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.382765 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.334847 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-var-lib-openvswitch\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.382765 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.334861 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-cni-binary-copy\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.382765 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.334883 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06f28cba-3757-4887-901d-d7b255673b78-system-cni-dir\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.382765 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.334934 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-host-var-lib-cni-multus\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.382765 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.334980 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-var-lib-kubelet\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.382765 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.334999 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-modprobe-d\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.382765 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.335027 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-host-var-lib-cni-multus\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.382765 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.335144 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-sysctl-conf\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.382765 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.335234 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-sysctl-conf\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.382765 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.335271 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-ovn-node-metrics-cert\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.382765 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.335298 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/06f28cba-3757-4887-901d-d7b255673b78-os-release\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.382765 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.335349 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwxrn\" (UniqueName: \"kubernetes.io/projected/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-kube-api-access-zwxrn\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.382765 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.335385 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f935f581-e0c8-48ec-aebb-0fd2444902d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.384103 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.335420 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f935f581-e0c8-48ec-aebb-0fd2444902d9-registration-dir\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.384103 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:31.335439 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:31.384103 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.335482 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f935f581-e0c8-48ec-aebb-0fd2444902d9-registration-dir\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.384103 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:31.335746 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs podName:848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c nodeName:}" failed. No retries permitted until 2026-07-07 19:00:31.835705745 +0000 UTC m=+2.028245813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs") pod "network-metrics-daemon-rfz5t" (UID: "848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:31.384103 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.335751 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f935f581-e0c8-48ec-aebb-0fd2444902d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.384103 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.335753 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-modprobe-d\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.384103 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.335770 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f935f581-e0c8-48ec-aebb-0fd2444902d9-device-dir\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.384103 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.335829 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-etc-openvswitch\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.384103 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.335918 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-run-openvswitch\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.384103 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.335838 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f935f581-e0c8-48ec-aebb-0fd2444902d9-device-dir\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.384103 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.335945 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/06f28cba-3757-4887-901d-d7b255673b78-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.384103 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:31.335989 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:31.384103 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.336002 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-multus-daemon-config\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.384103 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:31.336249 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret podName:c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac nodeName:}" failed. No retries permitted until 2026-07-07 19:00:31.836203049 +0000 UTC m=+2.028743103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret") pod "global-pull-secret-syncer-n6cxp" (UID: "c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:31.384103 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.336299 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db094710-2848-4e1b-a04f-bc0733b68e4e-host-slash\") pod \"iptables-alerter-45jxl\" (UID: \"db094710-2848-4e1b-a04f-bc0733b68e4e\") " pod="openshift-network-operator/iptables-alerter-45jxl" Jul 07 19:00:31.384103 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.336334 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-host\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.384794 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.336366 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-cni-netd\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.384794 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.336399 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.384794 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.336452 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krg8g\" (UniqueName: \"kubernetes.io/projected/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-kube-api-access-krg8g\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.384794 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.336476 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa2994d8-22f9-4a87-92c2-71f47f44fd4e-agent-certs\") pod \"konnectivity-agent-hvj8p\" (UID: \"fa2994d8-22f9-4a87-92c2-71f47f44fd4e\") " pod="kube-system/konnectivity-agent-hvj8p" Jul 07 19:00:31.384794 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.336537 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-multus-cni-dir\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.384794 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.336592 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-os-release\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.384794 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.336626 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-multus-socket-dir-parent\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.384794 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.336626 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-multus-daemon-config\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.384794 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.336659 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-node-log\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.384794 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.336751 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db094710-2848-4e1b-a04f-bc0733b68e4e-host-slash\") pod \"iptables-alerter-45jxl\" (UID: \"db094710-2848-4e1b-a04f-bc0733b68e4e\") " pod="openshift-network-operator/iptables-alerter-45jxl" Jul 07 19:00:31.384794 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.336894 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-multus-socket-dir-parent\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.384794 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.336901 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-multus-cni-dir\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.384794 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.336961 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06f28cba-3757-4887-901d-d7b255673b78-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.384794 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.336989 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-os-release\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.384794 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337010 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db094710-2848-4e1b-a04f-bc0733b68e4e-iptables-alerter-script\") pod \"iptables-alerter-45jxl\" (UID: \"db094710-2848-4e1b-a04f-bc0733b68e4e\") " pod="openshift-network-operator/iptables-alerter-45jxl" Jul 07 19:00:31.384794 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337036 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-host\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.384794 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337043 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-sysctl-d\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.385847 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337089 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-run-ovn\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.385847 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337213 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-dbus\") pod \"global-pull-secret-syncer-n6cxp\" (UID: \"c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac\") " pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:31.385847 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337282 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-systemd\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.385847 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337339 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-tuned\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.385847 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337436 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-run-netns\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.385847 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337496 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f935f581-e0c8-48ec-aebb-0fd2444902d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.385847 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337531 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-system-cni-dir\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.385847 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337564 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-host-run-k8s-cni-cncf-io\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.385847 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337585 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db094710-2848-4e1b-a04f-bc0733b68e4e-iptables-alerter-script\") pod \"iptables-alerter-45jxl\" (UID: \"db094710-2848-4e1b-a04f-bc0733b68e4e\") " pod="openshift-network-operator/iptables-alerter-45jxl" Jul 07 19:00:31.385847 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337629 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-ovnkube-config\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.385847 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337661 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-dbus\") pod \"global-pull-secret-syncer-n6cxp\" (UID: \"c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac\") " pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:31.385847 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337668 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/06f28cba-3757-4887-901d-d7b255673b78-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.385847 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337710 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-host-run-k8s-cni-cncf-io\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.385847 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337718 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4bc4\" (UniqueName: \"kubernetes.io/projected/f935f581-e0c8-48ec-aebb-0fd2444902d9-kube-api-access-h4bc4\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.385847 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337751 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crxh5\" (UniqueName: \"kubernetes.io/projected/db094710-2848-4e1b-a04f-bc0733b68e4e-kube-api-access-crxh5\") pod \"iptables-alerter-45jxl\" (UID: \"db094710-2848-4e1b-a04f-bc0733b68e4e\") " pod="openshift-network-operator/iptables-alerter-45jxl" Jul 07 19:00:31.385847 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337798 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-lib-modules\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.385847 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337869 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-env-overrides\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.386513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.337899 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa2994d8-22f9-4a87-92c2-71f47f44fd4e-konnectivity-ca\") pod \"konnectivity-agent-hvj8p\" (UID: \"fa2994d8-22f9-4a87-92c2-71f47f44fd4e\") " pod="kube-system/konnectivity-agent-hvj8p" Jul 07 19:00:31.386513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338049 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 19:00:31.386513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338181 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-sysctl-d\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.386513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338273 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-systemd-units\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.386513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338345 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61a2f3ec-3a02-49c4-984b-ca722d4f7914-host\") pod \"node-ca-vrwmc\" (UID: \"61a2f3ec-3a02-49c4-984b-ca722d4f7914\") " pod="openshift-image-registry/node-ca-vrwmc" Jul 07 19:00:31.386513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338402 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-systemd\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.386513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338410 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm4dv\" (UniqueName: \"kubernetes.io/projected/61a2f3ec-3a02-49c4-984b-ca722d4f7914-kube-api-access-mm4dv\") pod \"node-ca-vrwmc\" (UID: \"61a2f3ec-3a02-49c4-984b-ca722d4f7914\") " pod="openshift-image-registry/node-ca-vrwmc" Jul 07 19:00:31.386513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338414 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-lib-modules\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.386513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338252 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-system-cni-dir\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.386513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338480 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06f28cba-3757-4887-901d-d7b255673b78-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.386513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-hostroot\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.386513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338647 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f935f581-e0c8-48ec-aebb-0fd2444902d9-socket-dir\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.386513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338662 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f935f581-e0c8-48ec-aebb-0fd2444902d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.386513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338716 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-hostroot\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.386513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338717 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-slash\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.386513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338751 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-run-systemd\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.386513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338788 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f935f581-e0c8-48ec-aebb-0fd2444902d9-socket-dir\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.386513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338878 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-log-socket\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.387633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338913 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-run-ovn-kubernetes\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.387633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338942 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-host-var-lib-kubelet\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.387633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.338992 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-host-var-lib-kubelet\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.387633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339070 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-multus-conf-dir\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.387633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339113 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lwg9\" (UniqueName: \"kubernetes.io/projected/a079670f-9732-4279-901b-5fc3310a9a84-kube-api-access-2lwg9\") pod \"node-resolver-ptk5w\" (UID: \"a079670f-9732-4279-901b-5fc3310a9a84\") " pod="openshift-dns/node-resolver-ptk5w" Jul 07 19:00:31.387633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339166 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-multus-conf-dir\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.387633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339195 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-tmp\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.387633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339239 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8dlk\" (UniqueName: \"kubernetes.io/projected/06f28cba-3757-4887-901d-d7b255673b78-kube-api-access-c8dlk\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.387633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339311 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-host-run-netns\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.387633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339364 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-host-run-multus-certs\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.387633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339411 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-run\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.387633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339466 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-host-run-netns\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.387633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339503 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-host-run-multus-certs\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.387633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339514 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-sys\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.387633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339588 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-sys\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.387633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339633 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-run\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.387633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339654 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-ovnkube-script-lib\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.387633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339720 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f935f581-e0c8-48ec-aebb-0fd2444902d9-sys-fs\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.388375 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339771 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c7pr\" (UniqueName: \"kubernetes.io/projected/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-kube-api-access-5c7pr\") pod \"network-metrics-daemon-rfz5t\" (UID: \"848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c\") " pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:31.388375 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339806 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8g6k\" (UniqueName: \"kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k\") pod \"network-check-target-8m6sh\" (UID: \"1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b\") " pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:31.388375 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339830 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f935f581-e0c8-48ec-aebb-0fd2444902d9-sys-fs\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.388375 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339841 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-cni-bin\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.388375 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339872 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61a2f3ec-3a02-49c4-984b-ca722d4f7914-serviceca\") pod \"node-ca-vrwmc\" (UID: \"61a2f3ec-3a02-49c4-984b-ca722d4f7914\") " pod="openshift-image-registry/node-ca-vrwmc" Jul 07 19:00:31.388375 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.339966 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06f28cba-3757-4887-901d-d7b255673b78-cnibin\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.388375 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.342938 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-tmp\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.388375 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.344832 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n55d\" (UniqueName: \"kubernetes.io/projected/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-kube-api-access-4n55d\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.388375 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.344847 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwxrn\" (UniqueName: \"kubernetes.io/projected/3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a-kube-api-access-zwxrn\") pod \"multus-drmdz\" (UID: \"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a\") " pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.388375 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.348014 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crxh5\" (UniqueName: \"kubernetes.io/projected/db094710-2848-4e1b-a04f-bc0733b68e4e-kube-api-access-crxh5\") pod \"iptables-alerter-45jxl\" (UID: \"db094710-2848-4e1b-a04f-bc0733b68e4e\") " pod="openshift-network-operator/iptables-alerter-45jxl" Jul 07 19:00:31.388375 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.348107 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4bc4\" (UniqueName: \"kubernetes.io/projected/f935f581-e0c8-48ec-aebb-0fd2444902d9-kube-api-access-h4bc4\") pod \"aws-ebs-csi-driver-node-8zdpd\" (UID: \"f935f581-e0c8-48ec-aebb-0fd2444902d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.388375 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.351017 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lwg9\" (UniqueName: \"kubernetes.io/projected/a079670f-9732-4279-901b-5fc3310a9a84-kube-api-access-2lwg9\") pod \"node-resolver-ptk5w\" (UID: \"a079670f-9732-4279-901b-5fc3310a9a84\") " pod="openshift-dns/node-resolver-ptk5w" Jul 07 19:00:31.388375 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.352499 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c7pr\" (UniqueName: \"kubernetes.io/projected/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-kube-api-access-5c7pr\") pod \"network-metrics-daemon-rfz5t\" (UID: \"848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c\") " pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:31.388375 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.355235 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fd3d6596-5d49-4e7d-8e64-b5c3b30863bb-etc-tuned\") pod \"tuned-ldtzz\" (UID: \"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb\") " pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.391072 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:31.390716 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f9b3936a5fcd68b0bd917da8308fa9.slice/crio-bc65209519eda95f800707fb1f4a68bdad44bba31a4a22f1a647640daa4e730c WatchSource:0}: Error finding container bc65209519eda95f800707fb1f4a68bdad44bba31a4a22f1a647640daa4e730c: Status 404 returned error can't find the container with id bc65209519eda95f800707fb1f4a68bdad44bba31a4a22f1a647640daa4e730c Jul 07 19:00:31.391072 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:31.391047 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bad345f280d0902b6e7e2ae30ad5b4d.slice/crio-cf8d505cdc267de301a75f352ee8e91b45580eece3a6a2e2f2d42b27deb0bfdd WatchSource:0}: Error finding container cf8d505cdc267de301a75f352ee8e91b45580eece3a6a2e2f2d42b27deb0bfdd: Status 404 returned error can't find the container with id cf8d505cdc267de301a75f352ee8e91b45580eece3a6a2e2f2d42b27deb0bfdd Jul 07 19:00:31.504351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.403042 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:00:31.504351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.405071 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" Jul 07 19:00:31.504351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.440798 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-node-log\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.504351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.440827 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06f28cba-3757-4887-901d-d7b255673b78-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.504351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.440854 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-run-ovn\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.504351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.440871 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-run-netns\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.504351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.440889 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-ovnkube-config\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.504351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.440904 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/06f28cba-3757-4887-901d-d7b255673b78-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.504351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.440931 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-env-overrides\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.504351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.440947 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa2994d8-22f9-4a87-92c2-71f47f44fd4e-konnectivity-ca\") pod \"konnectivity-agent-hvj8p\" (UID: \"fa2994d8-22f9-4a87-92c2-71f47f44fd4e\") " pod="kube-system/konnectivity-agent-hvj8p" Jul 07 19:00:31.504351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.440962 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-systemd-units\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.504351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.440977 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61a2f3ec-3a02-49c4-984b-ca722d4f7914-host\") pod \"node-ca-vrwmc\" (UID: \"61a2f3ec-3a02-49c4-984b-ca722d4f7914\") " pod="openshift-image-registry/node-ca-vrwmc" Jul 07 19:00:31.504351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.440992 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm4dv\" (UniqueName: \"kubernetes.io/projected/61a2f3ec-3a02-49c4-984b-ca722d4f7914-kube-api-access-mm4dv\") pod \"node-ca-vrwmc\" (UID: \"61a2f3ec-3a02-49c4-984b-ca722d4f7914\") " pod="openshift-image-registry/node-ca-vrwmc" Jul 07 19:00:31.504351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441008 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06f28cba-3757-4887-901d-d7b255673b78-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.504351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441025 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-slash\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.504351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441045 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-run-systemd\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.504351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441060 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-log-socket\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.504351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441098 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-run-ovn-kubernetes\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.506473 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441118 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8dlk\" (UniqueName: \"kubernetes.io/projected/06f28cba-3757-4887-901d-d7b255673b78-kube-api-access-c8dlk\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.506473 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441136 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-ovnkube-script-lib\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.506473 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441160 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8g6k\" (UniqueName: \"kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k\") pod \"network-check-target-8m6sh\" (UID: \"1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b\") " pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:31.506473 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441176 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-cni-bin\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.506473 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441193 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61a2f3ec-3a02-49c4-984b-ca722d4f7914-serviceca\") pod \"node-ca-vrwmc\" (UID: \"61a2f3ec-3a02-49c4-984b-ca722d4f7914\") " pod="openshift-image-registry/node-ca-vrwmc" Jul 07 19:00:31.506473 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441209 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06f28cba-3757-4887-901d-d7b255673b78-cnibin\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.506473 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441234 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-kubelet\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.506473 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441286 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-var-lib-openvswitch\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.506473 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441301 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06f28cba-3757-4887-901d-d7b255673b78-system-cni-dir\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.506473 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441321 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-ovn-node-metrics-cert\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.506473 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441340 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/06f28cba-3757-4887-901d-d7b255673b78-os-release\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.506473 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441361 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-etc-openvswitch\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.506473 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441379 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-run-openvswitch\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.506473 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441397 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/06f28cba-3757-4887-901d-d7b255673b78-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.506473 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441416 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-cni-netd\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.506473 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441438 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.506473 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441455 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krg8g\" (UniqueName: \"kubernetes.io/projected/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-kube-api-access-krg8g\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.441476 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa2994d8-22f9-4a87-92c2-71f47f44fd4e-agent-certs\") pod \"konnectivity-agent-hvj8p\" (UID: \"fa2994d8-22f9-4a87-92c2-71f47f44fd4e\") " pod="kube-system/konnectivity-agent-hvj8p" Jul 07 19:00:31.507105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.443824 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa2994d8-22f9-4a87-92c2-71f47f44fd4e-agent-certs\") pod \"konnectivity-agent-hvj8p\" (UID: \"fa2994d8-22f9-4a87-92c2-71f47f44fd4e\") " pod="kube-system/konnectivity-agent-hvj8p" Jul 07 19:00:31.507105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.443893 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-node-log\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.444265 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06f28cba-3757-4887-901d-d7b255673b78-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.507105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.444438 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-run-ovn\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.444469 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-run-netns\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.444790 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-ovnkube-config\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.444902 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-cni-bin\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445015 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-var-lib-openvswitch\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445034 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06f28cba-3757-4887-901d-d7b255673b78-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.507105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445091 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-systemd-units\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445163 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61a2f3ec-3a02-49c4-984b-ca722d4f7914-host\") pod \"node-ca-vrwmc\" (UID: \"61a2f3ec-3a02-49c4-984b-ca722d4f7914\") " pod="openshift-image-registry/node-ca-vrwmc" Jul 07 19:00:31.507105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445317 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445357 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/06f28cba-3757-4887-901d-d7b255673b78-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.507105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445410 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-run-ovn-kubernetes\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445414 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-cni-netd\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445422 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-kubelet\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507755 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445463 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-log-socket\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507755 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445513 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/06f28cba-3757-4887-901d-d7b255673b78-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.507755 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445527 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/61a2f3ec-3a02-49c4-984b-ca722d4f7914-serviceca\") pod \"node-ca-vrwmc\" (UID: \"61a2f3ec-3a02-49c4-984b-ca722d4f7914\") " pod="openshift-image-registry/node-ca-vrwmc" Jul 07 19:00:31.507755 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445582 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-host-slash\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507755 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445594 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06f28cba-3757-4887-901d-d7b255673b78-cnibin\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.507755 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445615 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06f28cba-3757-4887-901d-d7b255673b78-system-cni-dir\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.507755 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445657 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/06f28cba-3757-4887-901d-d7b255673b78-os-release\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.507755 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445684 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-run-openvswitch\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507755 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445718 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-etc-openvswitch\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507755 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445718 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-run-systemd\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507755 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445781 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa2994d8-22f9-4a87-92c2-71f47f44fd4e-konnectivity-ca\") pod \"konnectivity-agent-hvj8p\" (UID: \"fa2994d8-22f9-4a87-92c2-71f47f44fd4e\") " pod="kube-system/konnectivity-agent-hvj8p" Jul 07 19:00:31.507755 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.445859 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-env-overrides\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507755 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.446914 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-ovnkube-script-lib\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507755 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.447074 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-ovn-node-metrics-cert\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.507755 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:31.455525 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:00:31.507755 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:31.455542 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:00:31.507755 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:31.455551 2568 projected.go:196] Error preparing data for projected volume kube-api-access-c8g6k for pod openshift-network-diagnostics/network-check-target-8m6sh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:31.508395 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:31.455599 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k podName:1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b nodeName:}" failed. No retries permitted until 2026-07-07 19:00:31.955585675 +0000 UTC m=+2.148125721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-c8g6k" (UniqueName: "kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k") pod "network-check-target-8m6sh" (UID: "1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:31.508395 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.457728 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm4dv\" (UniqueName: \"kubernetes.io/projected/61a2f3ec-3a02-49c4-984b-ca722d4f7914-kube-api-access-mm4dv\") pod \"node-ca-vrwmc\" (UID: \"61a2f3ec-3a02-49c4-984b-ca722d4f7914\") " pod="openshift-image-registry/node-ca-vrwmc" Jul 07 19:00:31.508395 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.457909 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krg8g\" (UniqueName: \"kubernetes.io/projected/8e2c0f4e-48ec-4049-bdff-1e9b8052f324-kube-api-access-krg8g\") pod \"ovnkube-node-sstqk\" (UID: \"8e2c0f4e-48ec-4049-bdff-1e9b8052f324\") " pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.508395 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.458087 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8dlk\" (UniqueName: \"kubernetes.io/projected/06f28cba-3757-4887-901d-d7b255673b78-kube-api-access-c8dlk\") pod \"multus-additional-cni-plugins-g2vp4\" (UID: \"06f28cba-3757-4887-901d-d7b255673b78\") " pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.620629 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.563547 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-drmdz" Jul 07 19:00:31.620629 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:31.572755 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df9ce5c_1c17_4c80_8c4c_aa5f656cd84a.slice/crio-4d7a5c41fb723791a89e9628d708f00351459b7aa78a825c454825c4803d9066 WatchSource:0}: Error finding container 4d7a5c41fb723791a89e9628d708f00351459b7aa78a825c454825c4803d9066: Status 404 returned error can't find the container with id 4d7a5c41fb723791a89e9628d708f00351459b7aa78a825c454825c4803d9066 Jul 07 19:00:31.620629 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.588218 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:00:31.620629 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.601237 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" Jul 07 19:00:31.620629 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:31.609834 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf935f581_e0c8_48ec_aebb_0fd2444902d9.slice/crio-a557dc3a546b96207aaaaf4d0289b00fe9d8e79ac43fe1af434921b04ae8a9ef WatchSource:0}: Error finding container a557dc3a546b96207aaaaf4d0289b00fe9d8e79ac43fe1af434921b04ae8a9ef: Status 404 returned error can't find the container with id a557dc3a546b96207aaaaf4d0289b00fe9d8e79ac43fe1af434921b04ae8a9ef Jul 07 19:00:31.620629 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.611327 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ptk5w" Jul 07 19:00:31.746676 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:31.626384 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda079670f_9732_4279_901b_5fc3310a9a84.slice/crio-2d0c8be02ca046c07ecaf7653d8e274bd07f3310013d29172c8e79f7d5f0a272 WatchSource:0}: Error finding container 2d0c8be02ca046c07ecaf7653d8e274bd07f3310013d29172c8e79f7d5f0a272: Status 404 returned error can't find the container with id 2d0c8be02ca046c07ecaf7653d8e274bd07f3310013d29172c8e79f7d5f0a272 Jul 07 19:00:31.746676 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.635548 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-45jxl" Jul 07 19:00:31.746676 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:31.645558 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb094710_2848_4e1b_a04f_bc0733b68e4e.slice/crio-9ed8dfeea5ddbc81abaa3e9f3625d662219bdfdbd97aa49014064b6d1f60c560 WatchSource:0}: Error finding container 9ed8dfeea5ddbc81abaa3e9f3625d662219bdfdbd97aa49014064b6d1f60c560: Status 404 returned error can't find the container with id 9ed8dfeea5ddbc81abaa3e9f3625d662219bdfdbd97aa49014064b6d1f60c560 Jul 07 19:00:31.746676 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.716552 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vrwmc" Jul 07 19:00:31.746676 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.731404 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g2vp4" Jul 07 19:00:31.746676 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.734615 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:31.746676 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:31.739160 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f28cba_3757_4887_901d_d7b255673b78.slice/crio-f084232745e6e8641cf6097f3e4ab7de419c2296d696e3c9f9ef733428311b7f WatchSource:0}: Error finding container f084232745e6e8641cf6097f3e4ab7de419c2296d696e3c9f9ef733428311b7f: Status 404 returned error can't find the container with id f084232745e6e8641cf6097f3e4ab7de419c2296d696e3c9f9ef733428311b7f Jul 07 19:00:31.746676 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.739800 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hvj8p" Jul 07 19:00:31.899326 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:00:31.750331 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2994d8_22f9_4a87_92c2_71f47f44fd4e.slice/crio-8e512e83792f8854f660b2d5c6bef79909a12a89c9ceb01d7f29eb88323d65c3 WatchSource:0}: Error finding container 8e512e83792f8854f660b2d5c6bef79909a12a89c9ceb01d7f29eb88323d65c3: Status 404 returned error can't find the container with id 8e512e83792f8854f660b2d5c6bef79909a12a89c9ceb01d7f29eb88323d65c3 Jul 07 19:00:31.899326 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.797164 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:00:31.899326 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.844037 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret\") pod \"global-pull-secret-syncer-n6cxp\" (UID: \"c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac\") " pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:31.899326 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:31.844080 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs\") pod \"network-metrics-daemon-rfz5t\" (UID: \"848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c\") " pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:31.899326 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:31.844183 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:31.899326 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:31.844239 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret podName:c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac nodeName:}" failed. No retries permitted until 2026-07-07 19:00:32.844223611 +0000 UTC m=+3.036763662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret") pod "global-pull-secret-syncer-n6cxp" (UID: "c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:31.899326 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:31.844183 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:31.899326 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:31.844307 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs podName:848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c nodeName:}" failed. No retries permitted until 2026-07-07 19:00:32.844294741 +0000 UTC m=+3.036834789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs") pod "network-metrics-daemon-rfz5t" (UID: "848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:32.080238 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:32.044801 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8g6k\" (UniqueName: \"kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k\") pod \"network-check-target-8m6sh\" (UID: \"1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b\") " pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:32.080238 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:32.044948 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:00:32.080238 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:32.044977 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:00:32.080238 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:32.044986 2568 projected.go:196] Error preparing data for projected volume kube-api-access-c8g6k for pod openshift-network-diagnostics/network-check-target-8m6sh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:32.080238 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:32.045037 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k podName:1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b nodeName:}" failed. No retries permitted until 2026-07-07 19:00:33.045021984 +0000 UTC m=+3.237562036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-c8g6k" (UniqueName: "kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k") pod "network-check-target-8m6sh" (UID: "1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:32.256511 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:32.256431 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:00:32.284087 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:32.283950 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 18:55:31 +0000 UTC" deadline="2027-12-22 20:31:08.655272718 +0000 UTC" Jul 07 19:00:32.284087 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:32.284014 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12793h30m36.371263072s" Jul 07 19:00:32.343260 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:32.343185 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" event={"ID":"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb","Type":"ContainerStarted","Data":"0d9d0d3fe3492a1394265d2eb3cc8ddc27b216c4fc943d8a9624538fdb9e1515"} Jul 07 19:00:32.345698 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:32.345669 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-109.ec2.internal" event={"ID":"3bad345f280d0902b6e7e2ae30ad5b4d","Type":"ContainerStarted","Data":"cf8d505cdc267de301a75f352ee8e91b45580eece3a6a2e2f2d42b27deb0bfdd"} Jul 07 19:00:32.352027 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:32.351966 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vrwmc" event={"ID":"61a2f3ec-3a02-49c4-984b-ca722d4f7914","Type":"ContainerStarted","Data":"7ed05ff54e5c98a7ec1e16fe9d85b929aac9bd4670de64531fafb4032d1ab495"} Jul 07 19:00:32.360292 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:32.360259 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-45jxl" event={"ID":"db094710-2848-4e1b-a04f-bc0733b68e4e","Type":"ContainerStarted","Data":"9ed8dfeea5ddbc81abaa3e9f3625d662219bdfdbd97aa49014064b6d1f60c560"} Jul 07 19:00:32.362972 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:32.362942 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ptk5w" event={"ID":"a079670f-9732-4279-901b-5fc3310a9a84","Type":"ContainerStarted","Data":"2d0c8be02ca046c07ecaf7653d8e274bd07f3310013d29172c8e79f7d5f0a272"} Jul 07 19:00:32.366933 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:32.366907 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-drmdz" event={"ID":"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a","Type":"ContainerStarted","Data":"4d7a5c41fb723791a89e9628d708f00351459b7aa78a825c454825c4803d9066"} Jul 07 19:00:32.373059 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:32.373030 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal" event={"ID":"13f9b3936a5fcd68b0bd917da8308fa9","Type":"ContainerStarted","Data":"bc65209519eda95f800707fb1f4a68bdad44bba31a4a22f1a647640daa4e730c"} Jul 07 19:00:32.377258 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:32.377232 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hvj8p" event={"ID":"fa2994d8-22f9-4a87-92c2-71f47f44fd4e","Type":"ContainerStarted","Data":"8e512e83792f8854f660b2d5c6bef79909a12a89c9ceb01d7f29eb88323d65c3"} Jul 07 19:00:32.384550 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:32.384409 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" event={"ID":"8e2c0f4e-48ec-4049-bdff-1e9b8052f324","Type":"ContainerStarted","Data":"270b8bfde7f9065f56527f8f3e37323b8c6ef07a6800478da5d036b5773ac7ce"} Jul 07 19:00:32.387749 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:32.387722 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2vp4" event={"ID":"06f28cba-3757-4887-901d-d7b255673b78","Type":"ContainerStarted","Data":"f084232745e6e8641cf6097f3e4ab7de419c2296d696e3c9f9ef733428311b7f"} Jul 07 19:00:32.390679 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:32.390653 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" event={"ID":"f935f581-e0c8-48ec-aebb-0fd2444902d9","Type":"ContainerStarted","Data":"a557dc3a546b96207aaaaf4d0289b00fe9d8e79ac43fe1af434921b04ae8a9ef"} Jul 07 19:00:32.853313 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:32.852501 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret\") pod \"global-pull-secret-syncer-n6cxp\" (UID: \"c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac\") " pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:32.853313 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:32.852553 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs\") pod \"network-metrics-daemon-rfz5t\" (UID: \"848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c\") " pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:32.853313 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:32.852720 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:32.853313 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:32.852783 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs podName:848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c nodeName:}" failed. No retries permitted until 2026-07-07 19:00:34.852764207 +0000 UTC m=+5.045304260 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs") pod "network-metrics-daemon-rfz5t" (UID: "848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:32.853313 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:32.853222 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:32.853313 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:32.853273 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret podName:c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac nodeName:}" failed. No retries permitted until 2026-07-07 19:00:34.853257262 +0000 UTC m=+5.045797312 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret") pod "global-pull-secret-syncer-n6cxp" (UID: "c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:33.054542 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:33.054505 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8g6k\" (UniqueName: \"kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k\") pod \"network-check-target-8m6sh\" (UID: \"1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b\") " pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:33.054793 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:33.054775 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:00:33.054864 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:33.054800 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:00:33.054864 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:33.054812 2568 projected.go:196] Error preparing data for projected volume kube-api-access-c8g6k for pod openshift-network-diagnostics/network-check-target-8m6sh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:33.054971 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:33.054863 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k podName:1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b nodeName:}" failed. No retries permitted until 2026-07-07 19:00:35.054850608 +0000 UTC m=+5.247390654 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-c8g6k" (UniqueName: "kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k") pod "network-check-target-8m6sh" (UID: "1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:33.286600 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:33.286463 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 18:55:31 +0000 UTC" deadline="2028-03-04 11:54:06.647422937 +0000 UTC" Jul 07 19:00:33.286600 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:33.286507 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14536h53m33.360921168s" Jul 07 19:00:33.323463 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:33.323428 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:33.323653 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:33.323589 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfz5t" podUID="848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c" Jul 07 19:00:33.323653 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:33.323624 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:33.323796 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:33.323625 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:33.323796 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:33.323754 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m6sh" podUID="1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b" Jul 07 19:00:33.323957 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:33.323827 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n6cxp" podUID="c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac" Jul 07 19:00:34.870159 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:34.869750 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret\") pod \"global-pull-secret-syncer-n6cxp\" (UID: \"c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac\") " pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:34.870159 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:34.869814 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs\") pod \"network-metrics-daemon-rfz5t\" (UID: \"848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c\") " pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:34.870159 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:34.869969 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:34.870159 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:34.870050 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs podName:848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c nodeName:}" failed. No retries permitted until 2026-07-07 19:00:38.870030499 +0000 UTC m=+9.062570546 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs") pod "network-metrics-daemon-rfz5t" (UID: "848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:34.870876 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:34.870509 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:34.870876 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:34.870598 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret podName:c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac nodeName:}" failed. No retries permitted until 2026-07-07 19:00:38.870581002 +0000 UTC m=+9.063121054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret") pod "global-pull-secret-syncer-n6cxp" (UID: "c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:35.073205 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:35.073153 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8g6k\" (UniqueName: \"kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k\") pod \"network-check-target-8m6sh\" (UID: \"1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b\") " pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:35.073420 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:35.073389 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:00:35.073420 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:35.073413 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:00:35.073539 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:35.073428 2568 projected.go:196] Error preparing data for projected volume kube-api-access-c8g6k for pod openshift-network-diagnostics/network-check-target-8m6sh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:35.073613 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:35.073596 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k podName:1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b nodeName:}" failed. No retries permitted until 2026-07-07 19:00:39.073575484 +0000 UTC m=+9.266115533 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-c8g6k" (UniqueName: "kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k") pod "network-check-target-8m6sh" (UID: "1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:35.323548 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:35.323436 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:35.323793 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:35.323572 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m6sh" podUID="1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b" Jul 07 19:00:35.323793 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:35.323634 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:35.323793 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:35.323634 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:35.323793 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:35.323748 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n6cxp" podUID="c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac" Jul 07 19:00:35.324206 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:35.323854 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfz5t" podUID="848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c" Jul 07 19:00:37.322584 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:37.322546 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:37.322584 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:37.322578 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:37.323165 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:37.322554 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:37.323165 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:37.322677 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m6sh" podUID="1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b" Jul 07 19:00:37.323165 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:37.322771 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n6cxp" podUID="c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac" Jul 07 19:00:37.323165 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:37.322855 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfz5t" podUID="848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c" Jul 07 19:00:38.907865 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:38.907826 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret\") pod \"global-pull-secret-syncer-n6cxp\" (UID: \"c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac\") " pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:38.908360 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:38.907880 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs\") pod \"network-metrics-daemon-rfz5t\" (UID: \"848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c\") " pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:38.908360 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:38.908073 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:38.908360 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:38.908140 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs podName:848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c nodeName:}" failed. No retries permitted until 2026-07-07 19:00:46.90812096 +0000 UTC m=+17.100661010 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs") pod "network-metrics-daemon-rfz5t" (UID: "848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:38.908360 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:38.908148 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:38.908360 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:38.908209 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret podName:c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac nodeName:}" failed. No retries permitted until 2026-07-07 19:00:46.90819157 +0000 UTC m=+17.100731624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret") pod "global-pull-secret-syncer-n6cxp" (UID: "c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:39.109998 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:39.109910 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8g6k\" (UniqueName: \"kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k\") pod \"network-check-target-8m6sh\" (UID: \"1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b\") " pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:39.110169 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:39.110124 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:00:39.110169 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:39.110142 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:00:39.110169 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:39.110153 2568 projected.go:196] Error preparing data for projected volume kube-api-access-c8g6k for pod openshift-network-diagnostics/network-check-target-8m6sh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:39.110390 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:39.110210 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k podName:1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b nodeName:}" failed. No retries permitted until 2026-07-07 19:00:47.11019237 +0000 UTC m=+17.302732436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-c8g6k" (UniqueName: "kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k") pod "network-check-target-8m6sh" (UID: "1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:39.323303 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:39.323266 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:39.323469 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:39.323383 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:39.323469 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:39.323393 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m6sh" podUID="1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b" Jul 07 19:00:39.323469 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:39.323432 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:39.323643 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:39.323570 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfz5t" podUID="848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c" Jul 07 19:00:39.323700 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:39.323646 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n6cxp" podUID="c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac" Jul 07 19:00:41.323058 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:41.322629 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:41.323058 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:41.322758 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n6cxp" podUID="c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac" Jul 07 19:00:41.323058 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:41.322786 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:41.323058 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:41.322816 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:41.323058 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:41.322876 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfz5t" podUID="848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c" Jul 07 19:00:41.323058 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:41.322949 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m6sh" podUID="1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b" Jul 07 19:00:43.322960 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:43.322922 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:43.323416 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:43.323070 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfz5t" podUID="848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c" Jul 07 19:00:43.323548 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:43.323437 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:43.323615 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:43.323555 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n6cxp" podUID="c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac" Jul 07 19:00:43.323615 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:43.323593 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:43.323698 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:43.323667 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m6sh" podUID="1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b" Jul 07 19:00:45.322545 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:45.322505 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:45.322956 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:45.322506 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:45.322956 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:45.322628 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m6sh" podUID="1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b" Jul 07 19:00:45.322956 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:45.322695 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfz5t" podUID="848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c" Jul 07 19:00:45.322956 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:45.322505 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:45.322956 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:45.322791 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n6cxp" podUID="c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac" Jul 07 19:00:46.969710 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:46.969664 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret\") pod \"global-pull-secret-syncer-n6cxp\" (UID: \"c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac\") " pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:46.970274 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:46.969722 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs\") pod \"network-metrics-daemon-rfz5t\" (UID: \"848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c\") " pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:46.970274 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:46.969794 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:46.970274 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:46.969828 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:46.970274 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:46.969878 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret podName:c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac nodeName:}" failed. No retries permitted until 2026-07-07 19:01:02.969856592 +0000 UTC m=+33.162396655 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret") pod "global-pull-secret-syncer-n6cxp" (UID: "c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:00:46.970274 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:46.969902 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs podName:848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c nodeName:}" failed. No retries permitted until 2026-07-07 19:01:02.969892039 +0000 UTC m=+33.162432088 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs") pod "network-metrics-daemon-rfz5t" (UID: "848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:00:47.171355 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:47.171151 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8g6k\" (UniqueName: \"kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k\") pod \"network-check-target-8m6sh\" (UID: \"1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b\") " pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:47.171355 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:47.171353 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:00:47.171551 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:47.171380 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:00:47.171551 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:47.171396 2568 projected.go:196] Error preparing data for projected volume kube-api-access-c8g6k for pod openshift-network-diagnostics/network-check-target-8m6sh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:47.171551 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:47.171463 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k podName:1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b nodeName:}" failed. No retries permitted until 2026-07-07 19:01:03.1714421 +0000 UTC m=+33.363982148 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-c8g6k" (UniqueName: "kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k") pod "network-check-target-8m6sh" (UID: "1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:00:47.322926 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:47.322834 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:47.322926 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:47.322865 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:47.322926 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:47.322866 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:47.323207 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:47.322961 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n6cxp" podUID="c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac" Jul 07 19:00:47.323207 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:47.323066 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfz5t" podUID="848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c" Jul 07 19:00:47.323207 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:47.323130 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m6sh" podUID="1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b" Jul 07 19:00:49.323403 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:49.323368 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:49.323874 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:49.323418 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:49.323874 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:49.323493 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:49.323874 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:49.323504 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m6sh" podUID="1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b" Jul 07 19:00:49.323874 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:49.323588 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n6cxp" podUID="c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac" Jul 07 19:00:49.323874 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:49.323649 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfz5t" podUID="848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c" Jul 07 19:00:51.323322 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:51.323284 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:51.323790 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:51.323305 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:51.323790 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:51.323412 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m6sh" podUID="1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b" Jul 07 19:00:51.323790 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:51.323305 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:51.323790 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:51.323494 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n6cxp" podUID="c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac" Jul 07 19:00:51.323790 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:51.323609 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfz5t" podUID="848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c" Jul 07 19:00:52.457092 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:52.457065 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-109.ec2.internal" event={"ID":"3bad345f280d0902b6e7e2ae30ad5b4d","Type":"ContainerStarted","Data":"91aaa78303fb711ceec3cda13ec1258dabc6f64b3eb393f70c03c93373bef85d"} Jul 07 19:00:52.473245 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:52.473201 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-109.ec2.internal" podStartSLOduration=21.47318698 podStartE2EDuration="21.47318698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:52.473020487 +0000 UTC m=+22.665560556" watchObservedRunningTime="2026-07-07 19:00:52.47318698 +0000 UTC m=+22.665727046" Jul 07 19:00:53.323693 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.323407 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:53.323879 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.323412 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:53.323879 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:53.323758 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n6cxp" podUID="c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac" Jul 07 19:00:53.323879 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.323453 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:53.324036 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:53.323867 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfz5t" podUID="848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c" Jul 07 19:00:53.324036 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:53.323924 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m6sh" podUID="1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b" Jul 07 19:00:53.461173 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.461139 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vrwmc" event={"ID":"61a2f3ec-3a02-49c4-984b-ca722d4f7914","Type":"ContainerStarted","Data":"859cd2fbdfbbd83eb19ff2731b6a3ae3186a1573ff0b88c6ab26c1b0b153628d"} Jul 07 19:00:53.464184 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.464152 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-45jxl" event={"ID":"db094710-2848-4e1b-a04f-bc0733b68e4e","Type":"ContainerStarted","Data":"785682ee3459da70676edcbd381c9ed56376bc59253c1e0db89180fa2bac42b8"} Jul 07 19:00:53.466275 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.466247 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ptk5w" event={"ID":"a079670f-9732-4279-901b-5fc3310a9a84","Type":"ContainerStarted","Data":"039b8e91114aa94240c9adecfe59d44e13bf9f23e63b800936c2965d6269c02e"} Jul 07 19:00:53.468442 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.468414 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-drmdz" event={"ID":"3df9ce5c-1c17-4c80-8c4c-aa5f656cd84a","Type":"ContainerStarted","Data":"c30e5b409fe1fa2dfd399cf5715bb487210438c42f2351fdc0de6f36f0e10244"} Jul 07 19:00:53.470513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.470487 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal" event={"ID":"13f9b3936a5fcd68b0bd917da8308fa9","Type":"ContainerStarted","Data":"c2ce0e8bb40649b099e9eb5ba40982ef2d1eb400a959266b13a228de672540c7"} Jul 07 19:00:53.477846 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.477803 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vrwmc" podStartSLOduration=1.931802817 podStartE2EDuration="22.477790835s" totalImagesPullingTime="20.545988018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:53.477658445 +0000 UTC m=+23.670198512" watchObservedRunningTime="2026-07-07 19:00:53.477790835 +0000 UTC m=+23.670330902" Jul 07 19:00:53.488237 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.488211 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hvj8p" event={"ID":"fa2994d8-22f9-4a87-92c2-71f47f44fd4e","Type":"ContainerStarted","Data":"750ba7498c426762ddd2ae9c97d0c741d2c9965e8175c60e90acedc24fed30eb"} Jul 07 19:00:53.490806 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.490788 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:00:53.491173 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.491152 2568 generic.go:356] "Generic (PLEG): container finished" podID="8e2c0f4e-48ec-4049-bdff-1e9b8052f324" containerID="78145977a46748a38b9c0f56c3b9a777c0ac0b316a322d81811e6c56a982a86a" exitCode=1 Jul 07 19:00:53.491240 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.491220 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" event={"ID":"8e2c0f4e-48ec-4049-bdff-1e9b8052f324","Type":"ContainerStarted","Data":"931a49dff00593db97e6e532f1afcc81de77e8faf75251a737aefb9f96c46eda"} Jul 07 19:00:53.491301 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.491288 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" event={"ID":"8e2c0f4e-48ec-4049-bdff-1e9b8052f324","Type":"ContainerStarted","Data":"fa48221b889eb446a37f90ca2ff1b2bf5172e1a447c402beef1b9c370784f0c6"} Jul 07 19:00:53.491346 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.491325 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" event={"ID":"8e2c0f4e-48ec-4049-bdff-1e9b8052f324","Type":"ContainerStarted","Data":"bc70dde6da3e67d844566c888e804ace892e5ed00106f9d03b4af6a85702a6d6"} Jul 07 19:00:53.491384 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.491358 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" event={"ID":"8e2c0f4e-48ec-4049-bdff-1e9b8052f324","Type":"ContainerStarted","Data":"510243dce1e2ed7b610bb3454927a54127ff7eb170a2d8f6545c323d5245fad9"} Jul 07 19:00:53.491431 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.491393 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" event={"ID":"8e2c0f4e-48ec-4049-bdff-1e9b8052f324","Type":"ContainerDied","Data":"78145977a46748a38b9c0f56c3b9a777c0ac0b316a322d81811e6c56a982a86a"} Jul 07 19:00:53.491431 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.491427 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" event={"ID":"8e2c0f4e-48ec-4049-bdff-1e9b8052f324","Type":"ContainerStarted","Data":"ad3cb7ae47efb1b2aaaad453952310be7ecd44761b05f7f7e0a0a8f3933dc40a"} Jul 07 19:00:53.492471 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.492454 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2vp4" event={"ID":"06f28cba-3757-4887-901d-d7b255673b78","Type":"ContainerStarted","Data":"9077af786f3784c6f54c9b3026d419ef76d28c6cedefb4e023fd54ee2d2fe33c"} Jul 07 19:00:53.493750 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.493725 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" event={"ID":"f935f581-e0c8-48ec-aebb-0fd2444902d9","Type":"ContainerStarted","Data":"ef690756abf12d6e1b2bab6f44c227031fa316c3ce3b566dd706de75b75d35f9"} Jul 07 19:00:53.494983 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.494959 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" event={"ID":"fd3d6596-5d49-4e7d-8e64-b5c3b30863bb","Type":"ContainerStarted","Data":"cec56207242fda4a44fa41ffe41b5e4e204ae249a9a41ef4f24b000cee4e6754"} Jul 07 19:00:53.511390 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.511348 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-drmdz" podStartSLOduration=2.773049027 podStartE2EDuration="23.511335381s" totalImagesPullingTime="20.738286354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:53.510887827 +0000 UTC m=+23.703427896" watchObservedRunningTime="2026-07-07 19:00:53.511335381 +0000 UTC m=+23.703875483" Jul 07 19:00:53.546877 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.546832 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ptk5w" podStartSLOduration=2.901593443 podStartE2EDuration="23.54681763s" totalImagesPullingTime="20.645224187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:53.529156404 +0000 UTC m=+23.721696472" watchObservedRunningTime="2026-07-07 19:00:53.54681763 +0000 UTC m=+23.739357698" Jul 07 19:00:53.547125 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.546990 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-45jxl" podStartSLOduration=1.937825697 podStartE2EDuration="22.546984078s" totalImagesPullingTime="20.609158381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:53.546623116 +0000 UTC m=+23.739163184" watchObservedRunningTime="2026-07-07 19:00:53.546984078 +0000 UTC m=+23.739524145" Jul 07 19:00:53.585625 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.585534 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hvj8p" podStartSLOduration=2.095688299 podStartE2EDuration="22.585519761s" totalImagesPullingTime="20.489831462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:53.584754201 +0000 UTC m=+23.777294267" watchObservedRunningTime="2026-07-07 19:00:53.585519761 +0000 UTC m=+23.778059828" Jul 07 19:00:53.602228 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:53.602189 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ldtzz" podStartSLOduration=1.7475448089999999 podStartE2EDuration="22.602177728s" totalImagesPullingTime="20.854632919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:53.601848005 +0000 UTC m=+23.794388076" watchObservedRunningTime="2026-07-07 19:00:53.602177728 +0000 UTC m=+23.794717795" Jul 07 19:00:54.498761 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:54.498728 2568 generic.go:356] "Generic (PLEG): container finished" podID="13f9b3936a5fcd68b0bd917da8308fa9" containerID="c2ce0e8bb40649b099e9eb5ba40982ef2d1eb400a959266b13a228de672540c7" exitCode=0 Jul 07 19:00:54.499287 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:54.498814 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal" event={"ID":"13f9b3936a5fcd68b0bd917da8308fa9","Type":"ContainerDied","Data":"c2ce0e8bb40649b099e9eb5ba40982ef2d1eb400a959266b13a228de672540c7"} Jul 07 19:00:54.500373 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:54.500340 2568 generic.go:356] "Generic (PLEG): container finished" podID="06f28cba-3757-4887-901d-d7b255673b78" containerID="9077af786f3784c6f54c9b3026d419ef76d28c6cedefb4e023fd54ee2d2fe33c" exitCode=0 Jul 07 19:00:54.500493 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:54.500437 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2vp4" event={"ID":"06f28cba-3757-4887-901d-d7b255673b78","Type":"ContainerDied","Data":"9077af786f3784c6f54c9b3026d419ef76d28c6cedefb4e023fd54ee2d2fe33c"} Jul 07 19:00:54.870480 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:54.870453 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 19:00:55.304628 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:55.304537 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T19:00:54.870474559Z","UUID":"73537c87-2d41-4d22-916e-9ff2587c559d","Handler":null,"Name":"","Endpoint":""} Jul 07 19:00:55.306137 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:55.306118 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 19:00:55.306232 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:55.306157 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 19:00:55.322653 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:55.322629 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:55.322764 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:55.322657 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:55.322764 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:55.322668 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:55.322857 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:55.322765 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfz5t" podUID="848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c" Jul 07 19:00:55.323015 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:55.322918 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m6sh" podUID="1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b" Jul 07 19:00:55.323103 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:55.323012 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n6cxp" podUID="c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac" Jul 07 19:00:55.506285 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:55.505775 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal" event={"ID":"13f9b3936a5fcd68b0bd917da8308fa9","Type":"ContainerStarted","Data":"6a914a9c0a9abd7f5ead5623067eac7acafda963a0e7b871b86efd8b10bb4650"} Jul 07 19:00:55.509876 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:55.509827 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:00:55.510429 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:55.510404 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" event={"ID":"8e2c0f4e-48ec-4049-bdff-1e9b8052f324","Type":"ContainerStarted","Data":"7a2c01233a39c36ad400ebad38150ba02b4c71c7e86a66220d8e5abbea27089a"} Jul 07 19:00:55.512503 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:55.512480 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" event={"ID":"f935f581-e0c8-48ec-aebb-0fd2444902d9","Type":"ContainerStarted","Data":"c3d7fcf7695bf29c894c203882c5bb5577ffd3f3d95be98b8dc4625159425990"} Jul 07 19:00:55.523170 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:55.523137 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-109.ec2.internal" podStartSLOduration=24.523125203 podStartE2EDuration="24.523125203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:55.522782388 +0000 UTC m=+25.715322455" watchObservedRunningTime="2026-07-07 19:00:55.523125203 +0000 UTC m=+25.715665269" Jul 07 19:00:56.228751 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:56.228719 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hvj8p" Jul 07 19:00:56.229319 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:56.229295 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hvj8p" Jul 07 19:00:56.518073 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:56.517965 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" event={"ID":"f935f581-e0c8-48ec-aebb-0fd2444902d9","Type":"ContainerStarted","Data":"fcb83ddcd65aeaea3c5dc72e336f78d1474f6a88af4ecfa47f19559ab09847a1"} Jul 07 19:00:56.537616 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:56.537559 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8zdpd" podStartSLOduration=3.355644299 podStartE2EDuration="26.537538661s" totalImagesPullingTime="23.181894362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:56.537281409 +0000 UTC m=+26.729821477" watchObservedRunningTime="2026-07-07 19:00:56.537538661 +0000 UTC m=+26.730078728" Jul 07 19:00:57.323004 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:57.322969 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:57.323201 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:57.322975 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:57.323201 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:57.323103 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m6sh" podUID="1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b" Jul 07 19:00:57.323201 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:57.323117 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:57.323455 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:57.323227 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfz5t" podUID="848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c" Jul 07 19:00:57.323455 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:57.323312 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n6cxp" podUID="c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac" Jul 07 19:00:58.527365 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:58.527138 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:00:58.528210 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:58.527723 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" event={"ID":"8e2c0f4e-48ec-4049-bdff-1e9b8052f324","Type":"ContainerStarted","Data":"ee8813e0fe21f3bd24f60ece81a76fd5d8119b34c9f6c62e3b033cc82045f7fa"} Jul 07 19:00:58.528210 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:58.528025 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:58.528210 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:58.528110 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:58.528368 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:58.528213 2568 scope.go:117] "RemoveContainer" containerID="78145977a46748a38b9c0f56c3b9a777c0ac0b316a322d81811e6c56a982a86a" Jul 07 19:00:58.546780 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:58.546760 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:59.323266 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:59.323232 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:59.323266 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:59.323256 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:59.323488 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:59.323232 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:59.323488 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:59.323355 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n6cxp" podUID="c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac" Jul 07 19:00:59.323488 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:59.323437 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m6sh" podUID="1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b" Jul 07 19:00:59.323659 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:59.323556 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfz5t" podUID="848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c" Jul 07 19:00:59.534904 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:59.534874 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:00:59.535448 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:59.535316 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" event={"ID":"8e2c0f4e-48ec-4049-bdff-1e9b8052f324","Type":"ContainerStarted","Data":"82914e68844bc35bdc399bd94be5f3f2fb0728d60e7562713b9e69f85d07301a"} Jul 07 19:00:59.535724 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:59.535679 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:59.552922 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:59.552897 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:00:59.566075 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:59.566008 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" podStartSLOduration=7.778022385 podStartE2EDuration="28.56598764s" totalImagesPullingTime="20.787965255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:00:59.564579456 +0000 UTC m=+29.757119522" watchObservedRunningTime="2026-07-07 19:00:59.56598764 +0000 UTC m=+29.758527707" Jul 07 19:00:59.765256 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:59.765216 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hvj8p" Jul 07 19:00:59.765432 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:59.765377 2568 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 07 19:00:59.765979 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:59.765944 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hvj8p" Jul 07 19:00:59.962446 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:59.961898 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n6cxp"] Jul 07 19:00:59.962446 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:59.962021 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:00:59.962446 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:59.962138 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n6cxp" podUID="c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac" Jul 07 19:00:59.963403 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:59.963376 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rfz5t"] Jul 07 19:00:59.963560 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:59.963490 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:00:59.963639 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:59.963601 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfz5t" podUID="848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c" Jul 07 19:00:59.963873 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:59.963852 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8m6sh"] Jul 07 19:00:59.963956 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:00:59.963936 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:00:59.964038 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:00:59.964015 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m6sh" podUID="1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b" Jul 07 19:01:01.323081 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:01.323022 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:01:01.323081 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:01.323078 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:01:01.323501 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:01.323090 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:01:01.323501 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:01.323222 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m6sh" podUID="1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b" Jul 07 19:01:01.323501 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:01.323278 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n6cxp" podUID="c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac" Jul 07 19:01:01.323501 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:01.323355 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfz5t" podUID="848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c" Jul 07 19:01:01.541234 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:01.541201 2568 generic.go:356] "Generic (PLEG): container finished" podID="06f28cba-3757-4887-901d-d7b255673b78" containerID="5de7642124091b86cf67ee806de229f98ce560bb27d1f28120d006272bfb542a" exitCode=0 Jul 07 19:01:01.541376 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:01.541282 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2vp4" event={"ID":"06f28cba-3757-4887-901d-d7b255673b78","Type":"ContainerDied","Data":"5de7642124091b86cf67ee806de229f98ce560bb27d1f28120d006272bfb542a"} Jul 07 19:01:02.546287 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:02.546086 2568 generic.go:356] "Generic (PLEG): container finished" podID="06f28cba-3757-4887-901d-d7b255673b78" containerID="e89207f3ee53f49f0264c812b0cc1fdf8fb76c157e47f9f708d8a831674cd4e5" exitCode=0 Jul 07 19:01:02.546636 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:02.546162 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2vp4" event={"ID":"06f28cba-3757-4887-901d-d7b255673b78","Type":"ContainerDied","Data":"e89207f3ee53f49f0264c812b0cc1fdf8fb76c157e47f9f708d8a831674cd4e5"} Jul 07 19:01:02.985854 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:02.985814 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret\") pod \"global-pull-secret-syncer-n6cxp\" (UID: \"c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac\") " pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:01:02.985854 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:02.985851 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs\") pod \"network-metrics-daemon-rfz5t\" (UID: \"848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c\") " pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:01:02.986026 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:02.985957 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:01:02.986026 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:02.985968 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:01:02.986026 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:02.986024 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs podName:848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c nodeName:}" failed. No retries permitted until 2026-07-07 19:01:34.986007579 +0000 UTC m=+65.178547641 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs") pod "network-metrics-daemon-rfz5t" (UID: "848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:01:02.986194 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:02.986056 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret podName:c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac nodeName:}" failed. No retries permitted until 2026-07-07 19:01:34.986034516 +0000 UTC m=+65.178574562 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret") pod "global-pull-secret-syncer-n6cxp" (UID: "c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:01:03.187533 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:03.187491 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8g6k\" (UniqueName: \"kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k\") pod \"network-check-target-8m6sh\" (UID: \"1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b\") " pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:01:03.187696 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:03.187640 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:01:03.187696 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:03.187664 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:01:03.187696 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:03.187677 2568 projected.go:196] Error preparing data for projected volume kube-api-access-c8g6k for pod openshift-network-diagnostics/network-check-target-8m6sh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:01:03.187861 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:03.187743 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k podName:1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b nodeName:}" failed. No retries permitted until 2026-07-07 19:01:35.187724192 +0000 UTC m=+65.380264261 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-c8g6k" (UniqueName: "kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k") pod "network-check-target-8m6sh" (UID: "1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:01:03.323135 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:03.323104 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:01:03.323273 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:03.323144 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:01:03.323273 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:03.323115 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:01:03.323273 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:03.323221 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n6cxp" podUID="c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac" Jul 07 19:01:03.323273 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:03.323266 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m6sh" podUID="1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b" Jul 07 19:01:03.323390 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:03.323319 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfz5t" podUID="848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c" Jul 07 19:01:03.550789 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:03.550697 2568 generic.go:356] "Generic (PLEG): container finished" podID="06f28cba-3757-4887-901d-d7b255673b78" containerID="5007a43b524c8ac577a98e33eef66f79143efd0c5563bce0898934ef90c69dba" exitCode=0 Jul 07 19:01:03.550789 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:03.550749 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2vp4" event={"ID":"06f28cba-3757-4887-901d-d7b255673b78","Type":"ContainerDied","Data":"5007a43b524c8ac577a98e33eef66f79143efd0c5563bce0898934ef90c69dba"} Jul 07 19:01:05.323180 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:05.323144 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:01:05.323588 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:05.323187 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:01:05.323588 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:05.323185 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:01:05.323588 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:05.323254 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m6sh" podUID="1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b" Jul 07 19:01:05.323588 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:05.323365 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n6cxp" podUID="c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac" Jul 07 19:01:05.323588 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:05.323465 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rfz5t" podUID="848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c" Jul 07 19:01:06.114878 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.114851 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-109.ec2.internal" event="NodeReady" Jul 07 19:01:06.115122 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.114970 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 19:01:06.161805 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.161717 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hrh48"] Jul 07 19:01:06.165945 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.165917 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hrh48" Jul 07 19:01:06.166678 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.166649 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vz9jl"] Jul 07 19:01:06.168156 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.167992 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 19:01:06.168285 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.168188 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d4n46\"" Jul 07 19:01:06.168351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.168292 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 19:01:06.170550 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.170532 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vz9jl" Jul 07 19:01:06.172635 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.172496 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 19:01:06.172635 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.172513 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 19:01:06.172635 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.172582 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hwc8x\"" Jul 07 19:01:06.172856 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.172813 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 19:01:06.175239 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.175211 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hrh48"] Jul 07 19:01:06.180994 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.180961 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vz9jl"] Jul 07 19:01:06.308355 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.308314 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-tmp-dir\") pod \"dns-default-hrh48\" (UID: \"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74\") " pod="openshift-dns/dns-default-hrh48" Jul 07 19:01:06.308544 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.308370 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq69z\" (UniqueName: \"kubernetes.io/projected/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-kube-api-access-mq69z\") pod \"dns-default-hrh48\" (UID: \"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74\") " pod="openshift-dns/dns-default-hrh48" Jul 07 19:01:06.308544 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.308404 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls\") pod \"dns-default-hrh48\" (UID: \"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74\") " pod="openshift-dns/dns-default-hrh48" Jul 07 19:01:06.308544 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.308452 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-config-volume\") pod \"dns-default-hrh48\" (UID: \"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74\") " pod="openshift-dns/dns-default-hrh48" Jul 07 19:01:06.308544 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.308505 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqmsx\" (UniqueName: \"kubernetes.io/projected/08cae73a-9871-4cf5-a4d5-1045cc284553-kube-api-access-dqmsx\") pod \"ingress-canary-vz9jl\" (UID: \"08cae73a-9871-4cf5-a4d5-1045cc284553\") " pod="openshift-ingress-canary/ingress-canary-vz9jl" Jul 07 19:01:06.308544 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.308531 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08cae73a-9871-4cf5-a4d5-1045cc284553-cert\") pod \"ingress-canary-vz9jl\" (UID: \"08cae73a-9871-4cf5-a4d5-1045cc284553\") " pod="openshift-ingress-canary/ingress-canary-vz9jl" Jul 07 19:01:06.409117 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.409079 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq69z\" (UniqueName: \"kubernetes.io/projected/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-kube-api-access-mq69z\") pod \"dns-default-hrh48\" (UID: \"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74\") " pod="openshift-dns/dns-default-hrh48" Jul 07 19:01:06.409117 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.409121 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls\") pod \"dns-default-hrh48\" (UID: \"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74\") " pod="openshift-dns/dns-default-hrh48" Jul 07 19:01:06.409620 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.409154 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-config-volume\") pod \"dns-default-hrh48\" (UID: \"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74\") " pod="openshift-dns/dns-default-hrh48" Jul 07 19:01:06.409620 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.409191 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqmsx\" (UniqueName: \"kubernetes.io/projected/08cae73a-9871-4cf5-a4d5-1045cc284553-kube-api-access-dqmsx\") pod \"ingress-canary-vz9jl\" (UID: \"08cae73a-9871-4cf5-a4d5-1045cc284553\") " pod="openshift-ingress-canary/ingress-canary-vz9jl" Jul 07 19:01:06.409620 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:06.409256 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:01:06.409620 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:06.409320 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls podName:34b59a25-fd9d-4ed1-befe-7a9d4ad5db74 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:06.909300397 +0000 UTC m=+37.101840466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls") pod "dns-default-hrh48" (UID: "34b59a25-fd9d-4ed1-befe-7a9d4ad5db74") : secret "dns-default-metrics-tls" not found Jul 07 19:01:06.409620 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.409348 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08cae73a-9871-4cf5-a4d5-1045cc284553-cert\") pod \"ingress-canary-vz9jl\" (UID: \"08cae73a-9871-4cf5-a4d5-1045cc284553\") " pod="openshift-ingress-canary/ingress-canary-vz9jl" Jul 07 19:01:06.409620 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.409396 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-tmp-dir\") pod \"dns-default-hrh48\" (UID: \"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74\") " pod="openshift-dns/dns-default-hrh48" Jul 07 19:01:06.409884 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.409657 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-tmp-dir\") pod \"dns-default-hrh48\" (UID: \"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74\") " pod="openshift-dns/dns-default-hrh48" Jul 07 19:01:06.409884 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.409748 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-config-volume\") pod \"dns-default-hrh48\" (UID: \"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74\") " pod="openshift-dns/dns-default-hrh48" Jul 07 19:01:06.413696 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.413633 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08cae73a-9871-4cf5-a4d5-1045cc284553-cert\") pod \"ingress-canary-vz9jl\" (UID: \"08cae73a-9871-4cf5-a4d5-1045cc284553\") " pod="openshift-ingress-canary/ingress-canary-vz9jl" Jul 07 19:01:06.418709 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.418686 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq69z\" (UniqueName: \"kubernetes.io/projected/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-kube-api-access-mq69z\") pod \"dns-default-hrh48\" (UID: \"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74\") " pod="openshift-dns/dns-default-hrh48" Jul 07 19:01:06.418911 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.418888 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqmsx\" (UniqueName: \"kubernetes.io/projected/08cae73a-9871-4cf5-a4d5-1045cc284553-kube-api-access-dqmsx\") pod \"ingress-canary-vz9jl\" (UID: \"08cae73a-9871-4cf5-a4d5-1045cc284553\") " pod="openshift-ingress-canary/ingress-canary-vz9jl" Jul 07 19:01:06.486876 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.486827 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vz9jl" Jul 07 19:01:06.645478 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.645446 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vz9jl"] Jul 07 19:01:06.649403 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:01:06.649377 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08cae73a_9871_4cf5_a4d5_1045cc284553.slice/crio-a2dda939f67712cbb19b2bc791688fae549277f330b93d29dd3176c5acc599dd WatchSource:0}: Error finding container a2dda939f67712cbb19b2bc791688fae549277f330b93d29dd3176c5acc599dd: Status 404 returned error can't find the container with id a2dda939f67712cbb19b2bc791688fae549277f330b93d29dd3176c5acc599dd Jul 07 19:01:06.913522 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:06.913487 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls\") pod \"dns-default-hrh48\" (UID: \"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74\") " pod="openshift-dns/dns-default-hrh48" Jul 07 19:01:06.913725 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:06.913628 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:01:06.913725 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:06.913698 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls podName:34b59a25-fd9d-4ed1-befe-7a9d4ad5db74 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:07.913678639 +0000 UTC m=+38.106218705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls") pod "dns-default-hrh48" (UID: "34b59a25-fd9d-4ed1-befe-7a9d4ad5db74") : secret "dns-default-metrics-tls" not found Jul 07 19:01:07.322643 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:07.322551 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:01:07.322793 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:07.322554 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:01:07.323082 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:07.322564 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:01:07.325357 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:07.325333 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 19:01:07.325357 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:07.325348 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 19:01:07.325947 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:07.325850 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 19:01:07.325947 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:07.325895 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-84rbs\"" Jul 07 19:01:07.325947 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:07.325907 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6dffl\"" Jul 07 19:01:07.326139 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:07.325896 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 19:01:07.563717 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:07.563675 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vz9jl" event={"ID":"08cae73a-9871-4cf5-a4d5-1045cc284553","Type":"ContainerStarted","Data":"a2dda939f67712cbb19b2bc791688fae549277f330b93d29dd3176c5acc599dd"} Jul 07 19:01:07.923693 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:07.923658 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls\") pod \"dns-default-hrh48\" (UID: \"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74\") " pod="openshift-dns/dns-default-hrh48" Jul 07 19:01:07.923885 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:07.923801 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:01:07.923885 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:07.923870 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls podName:34b59a25-fd9d-4ed1-befe-7a9d4ad5db74 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:09.923851939 +0000 UTC m=+40.116391997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls") pod "dns-default-hrh48" (UID: "34b59a25-fd9d-4ed1-befe-7a9d4ad5db74") : secret "dns-default-metrics-tls" not found Jul 07 19:01:09.938987 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:09.938947 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls\") pod \"dns-default-hrh48\" (UID: \"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74\") " pod="openshift-dns/dns-default-hrh48" Jul 07 19:01:09.939385 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:09.939112 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:01:09.939385 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:09.939175 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls podName:34b59a25-fd9d-4ed1-befe-7a9d4ad5db74 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:13.939159616 +0000 UTC m=+44.131699661 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls") pod "dns-default-hrh48" (UID: "34b59a25-fd9d-4ed1-befe-7a9d4ad5db74") : secret "dns-default-metrics-tls" not found Jul 07 19:01:10.573070 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:10.572966 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vz9jl" event={"ID":"08cae73a-9871-4cf5-a4d5-1045cc284553","Type":"ContainerStarted","Data":"02fb06fd42bd479d7f443c2aca4c1192576e56ac3b7eecd3b21f0e724a1c5e1c"} Jul 07 19:01:10.575768 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:10.575742 2568 generic.go:356] "Generic (PLEG): container finished" podID="06f28cba-3757-4887-901d-d7b255673b78" containerID="f5d70da21c8d18ccdcacb08dd064a7230899802ae940c0a2ac817c3004b36425" exitCode=0 Jul 07 19:01:10.575900 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:10.575791 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2vp4" event={"ID":"06f28cba-3757-4887-901d-d7b255673b78","Type":"ContainerDied","Data":"f5d70da21c8d18ccdcacb08dd064a7230899802ae940c0a2ac817c3004b36425"} Jul 07 19:01:10.597628 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:10.597579 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vz9jl" podStartSLOduration=1.092287572 podStartE2EDuration="4.597563821s" totalImagesPullingTime="3.505276249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:10.59567637 +0000 UTC m=+40.788216438" watchObservedRunningTime="2026-07-07 19:01:10.597563821 +0000 UTC m=+40.790103888" Jul 07 19:01:11.581207 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:11.580982 2568 generic.go:356] "Generic (PLEG): container finished" podID="06f28cba-3757-4887-901d-d7b255673b78" containerID="1d45b12fde5f0cc7a9c89f68979924efa1ae181ac87e062d44d2c8ceab87869e" exitCode=0 Jul 07 19:01:11.581608 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:11.581073 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2vp4" event={"ID":"06f28cba-3757-4887-901d-d7b255673b78","Type":"ContainerDied","Data":"1d45b12fde5f0cc7a9c89f68979924efa1ae181ac87e062d44d2c8ceab87869e"} Jul 07 19:01:12.586916 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:12.586871 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2vp4" event={"ID":"06f28cba-3757-4887-901d-d7b255673b78","Type":"ContainerStarted","Data":"7a36e147f37fa8db5b74d691ca83167059510da6847c52dc4a090c1c9bd72ad4"} Jul 07 19:01:12.612617 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:12.612557 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g2vp4" podStartSLOduration=6.892347969 podStartE2EDuration="41.612537579s" totalImagesPullingTime="34.72018961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:12.611834079 +0000 UTC m=+42.804374146" watchObservedRunningTime="2026-07-07 19:01:12.612537579 +0000 UTC m=+42.805077648" Jul 07 19:01:13.970581 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:13.970541 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls\") pod \"dns-default-hrh48\" (UID: \"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74\") " pod="openshift-dns/dns-default-hrh48" Jul 07 19:01:13.971070 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:13.970727 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:01:13.971070 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:13.970819 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls podName:34b59a25-fd9d-4ed1-befe-7a9d4ad5db74 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:21.970797108 +0000 UTC m=+52.163337159 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls") pod "dns-default-hrh48" (UID: "34b59a25-fd9d-4ed1-befe-7a9d4ad5db74") : secret "dns-default-metrics-tls" not found Jul 07 19:01:22.021972 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:22.021931 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls\") pod \"dns-default-hrh48\" (UID: \"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74\") " pod="openshift-dns/dns-default-hrh48" Jul 07 19:01:22.022390 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:22.022071 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:01:22.022390 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:22.022123 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls podName:34b59a25-fd9d-4ed1-befe-7a9d4ad5db74 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:38.022109311 +0000 UTC m=+68.214649356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls") pod "dns-default-hrh48" (UID: "34b59a25-fd9d-4ed1-befe-7a9d4ad5db74") : secret "dns-default-metrics-tls" not found Jul 07 19:01:25.388802 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.388764 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rg2g5"] Jul 07 19:01:25.390815 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.390799 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rg2g5" Jul 07 19:01:25.392872 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.392849 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 07 19:01:25.392994 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.392934 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-8ws5p\"" Jul 07 19:01:25.393455 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.393438 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:01:25.404799 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.404777 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rg2g5"] Jul 07 19:01:25.502269 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.502237 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-j7h8x"] Jul 07 19:01:25.504489 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.504468 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.507136 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.507107 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 19:01:25.507252 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.507146 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-trpwk\"" Jul 07 19:01:25.507252 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.507147 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 19:01:25.508111 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.508095 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 07 19:01:25.508173 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.508141 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 07 19:01:25.510556 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.510535 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mz9pd"] Jul 07 19:01:25.512618 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.512598 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mz9pd" Jul 07 19:01:25.517644 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.517615 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 07 19:01:25.517776 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.517753 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 07 19:01:25.517859 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.517789 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 07 19:01:25.517859 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.517821 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vhjjm\"" Jul 07 19:01:25.517968 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.517909 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 07 19:01:25.518020 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.518000 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:01:25.519295 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.519268 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-j7h8x"] Jul 07 19:01:25.529192 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.529170 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mz9pd"] Jul 07 19:01:25.545849 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.545821 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt8w7\" (UniqueName: \"kubernetes.io/projected/88388786-b65f-4506-a3c5-808df99f9fa7-kube-api-access-nt8w7\") pod \"volume-data-source-validator-5d456fdbcd-rg2g5\" (UID: \"88388786-b65f-4506-a3c5-808df99f9fa7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rg2g5" Jul 07 19:01:25.605461 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.605425 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69d7bf5b4f-gns76"] Jul 07 19:01:25.607513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.607498 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.609630 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.609609 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 19:01:25.609737 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.609609 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 19:01:25.609737 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.609610 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8tx7z\"" Jul 07 19:01:25.609925 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.609908 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 19:01:25.615184 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.615163 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 19:01:25.622910 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.622885 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69d7bf5b4f-gns76"] Jul 07 19:01:25.646346 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.646267 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/32659884-db20-426c-be99-476c7718c18d-snapshots\") pod \"insights-operator-5c847dcfc4-j7h8x\" (UID: \"32659884-db20-426c-be99-476c7718c18d\") " pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.646492 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.646352 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt8w7\" (UniqueName: \"kubernetes.io/projected/88388786-b65f-4506-a3c5-808df99f9fa7-kube-api-access-nt8w7\") pod \"volume-data-source-validator-5d456fdbcd-rg2g5\" (UID: \"88388786-b65f-4506-a3c5-808df99f9fa7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rg2g5" Jul 07 19:01:25.646492 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.646377 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/32659884-db20-426c-be99-476c7718c18d-serving-cert\") pod \"insights-operator-5c847dcfc4-j7h8x\" (UID: \"32659884-db20-426c-be99-476c7718c18d\") " pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.646492 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.646412 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhvq4\" (UniqueName: \"kubernetes.io/projected/32659884-db20-426c-be99-476c7718c18d-kube-api-access-bhvq4\") pod \"insights-operator-5c847dcfc4-j7h8x\" (UID: \"32659884-db20-426c-be99-476c7718c18d\") " pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.646492 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.646441 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7fcae971-26d4-43f1-aa35-179c080f18bc-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-mz9pd\" (UID: \"7fcae971-26d4-43f1-aa35-179c080f18bc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mz9pd" Jul 07 19:01:25.646492 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.646468 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmq7q\" (UniqueName: \"kubernetes.io/projected/7fcae971-26d4-43f1-aa35-179c080f18bc-kube-api-access-bmq7q\") pod \"kube-storage-version-migrator-operator-846bf668f7-mz9pd\" (UID: \"7fcae971-26d4-43f1-aa35-179c080f18bc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mz9pd" Jul 07 19:01:25.646685 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.646512 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32659884-db20-426c-be99-476c7718c18d-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-j7h8x\" (UID: \"32659884-db20-426c-be99-476c7718c18d\") " pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.646685 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.646539 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32659884-db20-426c-be99-476c7718c18d-tmp\") pod \"insights-operator-5c847dcfc4-j7h8x\" (UID: \"32659884-db20-426c-be99-476c7718c18d\") " pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.646685 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.646593 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32659884-db20-426c-be99-476c7718c18d-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-j7h8x\" (UID: \"32659884-db20-426c-be99-476c7718c18d\") " pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.646685 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.646641 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7fcae971-26d4-43f1-aa35-179c080f18bc-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-mz9pd\" (UID: \"7fcae971-26d4-43f1-aa35-179c080f18bc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mz9pd" Jul 07 19:01:25.658099 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.658076 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt8w7\" (UniqueName: \"kubernetes.io/projected/88388786-b65f-4506-a3c5-808df99f9fa7-kube-api-access-nt8w7\") pod \"volume-data-source-validator-5d456fdbcd-rg2g5\" (UID: \"88388786-b65f-4506-a3c5-808df99f9fa7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rg2g5" Jul 07 19:01:25.698966 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.698937 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rg2g5" Jul 07 19:01:25.747436 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.747414 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhvq4\" (UniqueName: \"kubernetes.io/projected/32659884-db20-426c-be99-476c7718c18d-kube-api-access-bhvq4\") pod \"insights-operator-5c847dcfc4-j7h8x\" (UID: \"32659884-db20-426c-be99-476c7718c18d\") " pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.747629 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.747608 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7fcae971-26d4-43f1-aa35-179c080f18bc-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-mz9pd\" (UID: \"7fcae971-26d4-43f1-aa35-179c080f18bc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mz9pd" Jul 07 19:01:25.747756 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.747739 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10c78cf7-3156-415b-8278-bf57dd5290b0-ca-trust-extracted\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.747883 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.747866 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-certificates\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.748019 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.748002 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmq7q\" (UniqueName: \"kubernetes.io/projected/7fcae971-26d4-43f1-aa35-179c080f18bc-kube-api-access-bmq7q\") pod \"kube-storage-version-migrator-operator-846bf668f7-mz9pd\" (UID: \"7fcae971-26d4-43f1-aa35-179c080f18bc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mz9pd" Jul 07 19:01:25.748152 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.748137 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-bound-sa-token\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.748282 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.748267 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9d8m\" (UniqueName: \"kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-kube-api-access-h9d8m\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.748427 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.748412 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32659884-db20-426c-be99-476c7718c18d-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-j7h8x\" (UID: \"32659884-db20-426c-be99-476c7718c18d\") " pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.748562 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.748547 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32659884-db20-426c-be99-476c7718c18d-tmp\") pod \"insights-operator-5c847dcfc4-j7h8x\" (UID: \"32659884-db20-426c-be99-476c7718c18d\") " pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.748700 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.748685 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32659884-db20-426c-be99-476c7718c18d-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-j7h8x\" (UID: \"32659884-db20-426c-be99-476c7718c18d\") " pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.748821 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.748804 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10c78cf7-3156-415b-8278-bf57dd5290b0-image-registry-private-configuration\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.748949 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.748925 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32659884-db20-426c-be99-476c7718c18d-tmp\") pod \"insights-operator-5c847dcfc4-j7h8x\" (UID: \"32659884-db20-426c-be99-476c7718c18d\") " pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.749029 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.748443 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7fcae971-26d4-43f1-aa35-179c080f18bc-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-mz9pd\" (UID: \"7fcae971-26d4-43f1-aa35-179c080f18bc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mz9pd" Jul 07 19:01:25.749029 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.749018 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32659884-db20-426c-be99-476c7718c18d-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-j7h8x\" (UID: \"32659884-db20-426c-be99-476c7718c18d\") " pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.749156 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.749091 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10c78cf7-3156-415b-8278-bf57dd5290b0-trusted-ca\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.749156 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.749138 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10c78cf7-3156-415b-8278-bf57dd5290b0-installation-pull-secrets\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.749261 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.749245 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7fcae971-26d4-43f1-aa35-179c080f18bc-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-mz9pd\" (UID: \"7fcae971-26d4-43f1-aa35-179c080f18bc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mz9pd" Jul 07 19:01:25.749321 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.749306 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.749416 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.749402 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/32659884-db20-426c-be99-476c7718c18d-snapshots\") pod \"insights-operator-5c847dcfc4-j7h8x\" (UID: \"32659884-db20-426c-be99-476c7718c18d\") " pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.749595 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.749580 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/32659884-db20-426c-be99-476c7718c18d-serving-cert\") pod \"insights-operator-5c847dcfc4-j7h8x\" (UID: \"32659884-db20-426c-be99-476c7718c18d\") " pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.749736 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.749675 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32659884-db20-426c-be99-476c7718c18d-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-j7h8x\" (UID: \"32659884-db20-426c-be99-476c7718c18d\") " pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.749946 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.749921 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/32659884-db20-426c-be99-476c7718c18d-snapshots\") pod \"insights-operator-5c847dcfc4-j7h8x\" (UID: \"32659884-db20-426c-be99-476c7718c18d\") " pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.752061 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.752001 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7fcae971-26d4-43f1-aa35-179c080f18bc-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-mz9pd\" (UID: \"7fcae971-26d4-43f1-aa35-179c080f18bc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mz9pd" Jul 07 19:01:25.752159 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.752142 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/32659884-db20-426c-be99-476c7718c18d-serving-cert\") pod \"insights-operator-5c847dcfc4-j7h8x\" (UID: \"32659884-db20-426c-be99-476c7718c18d\") " pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.760397 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.759941 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmq7q\" (UniqueName: \"kubernetes.io/projected/7fcae971-26d4-43f1-aa35-179c080f18bc-kube-api-access-bmq7q\") pod \"kube-storage-version-migrator-operator-846bf668f7-mz9pd\" (UID: \"7fcae971-26d4-43f1-aa35-179c080f18bc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mz9pd" Jul 07 19:01:25.760397 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.760336 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhvq4\" (UniqueName: \"kubernetes.io/projected/32659884-db20-426c-be99-476c7718c18d-kube-api-access-bhvq4\") pod \"insights-operator-5c847dcfc4-j7h8x\" (UID: \"32659884-db20-426c-be99-476c7718c18d\") " pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.814100 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.814069 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" Jul 07 19:01:25.823833 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.823779 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rg2g5"] Jul 07 19:01:25.824098 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.824079 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mz9pd" Jul 07 19:01:25.828285 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:01:25.828253 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88388786_b65f_4506_a3c5_808df99f9fa7.slice/crio-e95fe00e07756d35ff9f462b585f68a84aa6b91b5f3d7d98a52e4e49c7217c42 WatchSource:0}: Error finding container e95fe00e07756d35ff9f462b585f68a84aa6b91b5f3d7d98a52e4e49c7217c42: Status 404 returned error can't find the container with id e95fe00e07756d35ff9f462b585f68a84aa6b91b5f3d7d98a52e4e49c7217c42 Jul 07 19:01:25.850569 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.850538 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10c78cf7-3156-415b-8278-bf57dd5290b0-image-registry-private-configuration\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.850739 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.850580 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10c78cf7-3156-415b-8278-bf57dd5290b0-trusted-ca\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.850739 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.850605 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10c78cf7-3156-415b-8278-bf57dd5290b0-installation-pull-secrets\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.850739 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.850654 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.850948 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.850745 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10c78cf7-3156-415b-8278-bf57dd5290b0-ca-trust-extracted\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.850948 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.850769 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-certificates\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.850948 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.850799 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-bound-sa-token\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.850948 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:25.850811 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 19:01:25.850948 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:25.850829 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69d7bf5b4f-gns76: secret "image-registry-tls" not found Jul 07 19:01:25.850948 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:25.850903 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls podName:10c78cf7-3156-415b-8278-bf57dd5290b0 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:26.350881253 +0000 UTC m=+56.543421316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls") pod "image-registry-69d7bf5b4f-gns76" (UID: "10c78cf7-3156-415b-8278-bf57dd5290b0") : secret "image-registry-tls" not found Jul 07 19:01:25.850948 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.850826 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9d8m\" (UniqueName: \"kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-kube-api-access-h9d8m\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.851913 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.851888 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-certificates\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.852087 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.851963 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10c78cf7-3156-415b-8278-bf57dd5290b0-trusted-ca\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.852243 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.852224 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10c78cf7-3156-415b-8278-bf57dd5290b0-ca-trust-extracted\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.853068 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.853025 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10c78cf7-3156-415b-8278-bf57dd5290b0-installation-pull-secrets\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.853658 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.853642 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10c78cf7-3156-415b-8278-bf57dd5290b0-image-registry-private-configuration\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.860878 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.860854 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-bound-sa-token\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.861551 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.861530 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9d8m\" (UniqueName: \"kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-kube-api-access-h9d8m\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:25.950565 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.950416 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-j7h8x"] Jul 07 19:01:25.952786 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:01:25.952761 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32659884_db20_426c_be99_476c7718c18d.slice/crio-972dacb905be0650e3386778ec24ca20c32ae36f2b3d42d123e944c28cffda08 WatchSource:0}: Error finding container 972dacb905be0650e3386778ec24ca20c32ae36f2b3d42d123e944c28cffda08: Status 404 returned error can't find the container with id 972dacb905be0650e3386778ec24ca20c32ae36f2b3d42d123e944c28cffda08 Jul 07 19:01:25.970983 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:25.970952 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mz9pd"] Jul 07 19:01:25.974333 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:01:25.974299 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcae971_26d4_43f1_aa35_179c080f18bc.slice/crio-16907b9d52ca9c162997794dea98fac392a8398779db84d5a6f097b733ec06fc WatchSource:0}: Error finding container 16907b9d52ca9c162997794dea98fac392a8398779db84d5a6f097b733ec06fc: Status 404 returned error can't find the container with id 16907b9d52ca9c162997794dea98fac392a8398779db84d5a6f097b733ec06fc Jul 07 19:01:26.354278 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:26.354192 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:26.354486 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:26.354326 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 19:01:26.354486 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:26.354346 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69d7bf5b4f-gns76: secret "image-registry-tls" not found Jul 07 19:01:26.354486 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:26.354399 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls podName:10c78cf7-3156-415b-8278-bf57dd5290b0 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:27.35438243 +0000 UTC m=+57.546922488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls") pod "image-registry-69d7bf5b4f-gns76" (UID: "10c78cf7-3156-415b-8278-bf57dd5290b0") : secret "image-registry-tls" not found Jul 07 19:01:26.620480 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:26.620396 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mz9pd" event={"ID":"7fcae971-26d4-43f1-aa35-179c080f18bc","Type":"ContainerStarted","Data":"16907b9d52ca9c162997794dea98fac392a8398779db84d5a6f097b733ec06fc"} Jul 07 19:01:26.621468 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:26.621448 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" event={"ID":"32659884-db20-426c-be99-476c7718c18d","Type":"ContainerStarted","Data":"972dacb905be0650e3386778ec24ca20c32ae36f2b3d42d123e944c28cffda08"} Jul 07 19:01:26.622493 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:26.622471 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rg2g5" event={"ID":"88388786-b65f-4506-a3c5-808df99f9fa7","Type":"ContainerStarted","Data":"e95fe00e07756d35ff9f462b585f68a84aa6b91b5f3d7d98a52e4e49c7217c42"} Jul 07 19:01:27.360615 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:27.360566 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:27.360807 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:27.360709 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 19:01:27.360807 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:27.360733 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69d7bf5b4f-gns76: secret "image-registry-tls" not found Jul 07 19:01:27.360807 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:27.360801 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls podName:10c78cf7-3156-415b-8278-bf57dd5290b0 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:29.360781355 +0000 UTC m=+59.553321407 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls") pod "image-registry-69d7bf5b4f-gns76" (UID: "10c78cf7-3156-415b-8278-bf57dd5290b0") : secret "image-registry-tls" not found Jul 07 19:01:27.626087 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:27.626030 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rg2g5" event={"ID":"88388786-b65f-4506-a3c5-808df99f9fa7","Type":"ContainerStarted","Data":"18531080ad2c51efc544eb9f2cecd50b9734d920353572af2a9650ac1a129930"} Jul 07 19:01:27.647896 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:27.647843 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-rg2g5" podStartSLOduration=1.352011754 podStartE2EDuration="2.647826067s" totalImagesPullingTime="1.295814313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:27.646198915 +0000 UTC m=+57.838738982" watchObservedRunningTime="2026-07-07 19:01:27.647826067 +0000 UTC m=+57.840366151" Jul 07 19:01:29.374748 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.374704 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:29.375153 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:29.374889 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 19:01:29.375153 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:29.374915 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69d7bf5b4f-gns76: secret "image-registry-tls" not found Jul 07 19:01:29.375153 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:29.374982 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls podName:10c78cf7-3156-415b-8278-bf57dd5290b0 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:33.374963906 +0000 UTC m=+63.567503957 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls") pod "image-registry-69d7bf5b4f-gns76" (UID: "10c78cf7-3156-415b-8278-bf57dd5290b0") : secret "image-registry-tls" not found Jul 07 19:01:29.635075 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.634750 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mz9pd" event={"ID":"7fcae971-26d4-43f1-aa35-179c080f18bc","Type":"ContainerStarted","Data":"9fab801178950cc4928dea2a002faa78b30f6e4df3deb9d176b38776514692d7"} Jul 07 19:01:29.636503 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.636476 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" event={"ID":"32659884-db20-426c-be99-476c7718c18d","Type":"ContainerStarted","Data":"19df1d61e18a4c2d6688280654ae92e88771e3a4a489f7420199f407d764b6ec"} Jul 07 19:01:29.720984 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.720945 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-trklp"] Jul 07 19:01:29.723774 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.723753 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:29.739829 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.739807 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bj6vq\"" Jul 07 19:01:29.744844 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.744819 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 19:01:29.752538 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.752522 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 19:01:29.776373 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.776348 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-trklp"] Jul 07 19:01:29.776877 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.776826 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mz9pd" podStartSLOduration=1.889984733 podStartE2EDuration="4.776811782s" totalImagesPullingTime="2.886827049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:29.774747566 +0000 UTC m=+59.967287633" watchObservedRunningTime="2026-07-07 19:01:29.776811782 +0000 UTC m=+59.969351898" Jul 07 19:01:29.777898 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.777870 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d200611b-531a-49e6-8d44-20ba3ac01a4a-crio-socket\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:29.777999 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.777909 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwl5r\" (UniqueName: \"kubernetes.io/projected/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-api-access-jwl5r\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:29.778114 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.778074 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:29.778114 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.778113 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d200611b-531a-49e6-8d44-20ba3ac01a4a-data-volume\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:29.778298 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.778229 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:29.817522 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.817469 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" podStartSLOduration=1.9114301729999998 podStartE2EDuration="4.817455139s" totalImagesPullingTime="2.906024966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:29.816263464 +0000 UTC m=+60.008803531" watchObservedRunningTime="2026-07-07 19:01:29.817455139 +0000 UTC m=+60.009995205" Jul 07 19:01:29.880141 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.880107 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d200611b-531a-49e6-8d44-20ba3ac01a4a-crio-socket\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:29.880291 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.880149 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwl5r\" (UniqueName: \"kubernetes.io/projected/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-api-access-jwl5r\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:29.880291 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.880232 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d200611b-531a-49e6-8d44-20ba3ac01a4a-crio-socket\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:29.880419 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.880333 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:29.880419 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.880353 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d200611b-531a-49e6-8d44-20ba3ac01a4a-data-volume\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:29.880419 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.880412 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:29.880555 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:29.880447 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 19:01:29.880555 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:29.880514 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-rbac-proxy-cm podName:d200611b-531a-49e6-8d44-20ba3ac01a4a nodeName:}" failed. No retries permitted until 2026-07-07 19:01:30.380497215 +0000 UTC m=+60.573037274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-trklp" (UID: "d200611b-531a-49e6-8d44-20ba3ac01a4a") : configmap "kube-rbac-proxy" not found Jul 07 19:01:29.880555 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:29.880525 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:01:29.880555 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:29.880558 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls podName:d200611b-531a-49e6-8d44-20ba3ac01a4a nodeName:}" failed. No retries permitted until 2026-07-07 19:01:30.380548355 +0000 UTC m=+60.573088415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-trklp" (UID: "d200611b-531a-49e6-8d44-20ba3ac01a4a") : secret "insights-runtime-extractor-tls" not found Jul 07 19:01:29.880811 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.880790 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d200611b-531a-49e6-8d44-20ba3ac01a4a-data-volume\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:29.890060 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:29.889986 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwl5r\" (UniqueName: \"kubernetes.io/projected/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-api-access-jwl5r\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:30.384392 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:30.384361 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:30.384755 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:30.384448 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:30.384755 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:30.384477 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 19:01:30.384755 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:30.384542 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-rbac-proxy-cm podName:d200611b-531a-49e6-8d44-20ba3ac01a4a nodeName:}" failed. No retries permitted until 2026-07-07 19:01:31.384526141 +0000 UTC m=+61.577066187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-trklp" (UID: "d200611b-531a-49e6-8d44-20ba3ac01a4a") : configmap "kube-rbac-proxy" not found Jul 07 19:01:30.384755 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:30.384571 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:01:30.384755 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:30.384626 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls podName:d200611b-531a-49e6-8d44-20ba3ac01a4a nodeName:}" failed. No retries permitted until 2026-07-07 19:01:31.384611248 +0000 UTC m=+61.577151300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-trklp" (UID: "d200611b-531a-49e6-8d44-20ba3ac01a4a") : secret "insights-runtime-extractor-tls" not found Jul 07 19:01:31.393765 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:31.393715 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:31.394211 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:31.393871 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:01:31.394211 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:31.393937 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls podName:d200611b-531a-49e6-8d44-20ba3ac01a4a nodeName:}" failed. No retries permitted until 2026-07-07 19:01:33.393916208 +0000 UTC m=+63.586456254 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-trklp" (UID: "d200611b-531a-49e6-8d44-20ba3ac01a4a") : secret "insights-runtime-extractor-tls" not found Jul 07 19:01:31.394211 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:31.393945 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 19:01:31.394211 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:31.393999 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-rbac-proxy-cm podName:d200611b-531a-49e6-8d44-20ba3ac01a4a nodeName:}" failed. No retries permitted until 2026-07-07 19:01:33.393987767 +0000 UTC m=+63.586527815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-trklp" (UID: "d200611b-531a-49e6-8d44-20ba3ac01a4a") : configmap "kube-rbac-proxy" not found Jul 07 19:01:31.394211 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:31.393874 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:31.562969 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:31.562945 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sstqk" Jul 07 19:01:31.648917 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:31.648835 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ptk5w_a079670f-9732-4279-901b-5fc3310a9a84/dns-node-resolver/0.log" Jul 07 19:01:32.172799 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:32.172747 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-6487b"] Jul 07 19:01:32.176510 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:32.176485 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6487b" Jul 07 19:01:32.178933 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:32.178914 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 07 19:01:32.179483 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:32.179467 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 07 19:01:32.179563 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:32.179547 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kc2p6\"" Jul 07 19:01:32.184788 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:32.184766 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-6487b"] Jul 07 19:01:32.200844 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:32.200809 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6487b\" (UID: \"cadb609a-f2e9-4ca0-a4eb-85952ff8160c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6487b" Jul 07 19:01:32.200970 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:32.200848 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-6487b\" (UID: \"cadb609a-f2e9-4ca0-a4eb-85952ff8160c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6487b" Jul 07 19:01:32.302111 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:32.302071 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6487b\" (UID: \"cadb609a-f2e9-4ca0-a4eb-85952ff8160c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6487b" Jul 07 19:01:32.302111 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:32.302113 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-6487b\" (UID: \"cadb609a-f2e9-4ca0-a4eb-85952ff8160c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6487b" Jul 07 19:01:32.302367 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:32.302228 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 19:01:32.302367 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:32.302354 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert podName:cadb609a-f2e9-4ca0-a4eb-85952ff8160c nodeName:}" failed. No retries permitted until 2026-07-07 19:01:32.802330914 +0000 UTC m=+62.994870961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-6487b" (UID: "cadb609a-f2e9-4ca0-a4eb-85952ff8160c") : secret "networking-console-plugin-cert" not found Jul 07 19:01:32.302790 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:32.302769 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-6487b\" (UID: \"cadb609a-f2e9-4ca0-a4eb-85952ff8160c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6487b" Jul 07 19:01:32.807055 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:32.807015 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6487b\" (UID: \"cadb609a-f2e9-4ca0-a4eb-85952ff8160c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6487b" Jul 07 19:01:32.807469 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:32.807164 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 19:01:32.807469 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:32.807243 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert podName:cadb609a-f2e9-4ca0-a4eb-85952ff8160c nodeName:}" failed. No retries permitted until 2026-07-07 19:01:33.807227401 +0000 UTC m=+63.999767449 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-6487b" (UID: "cadb609a-f2e9-4ca0-a4eb-85952ff8160c") : secret "networking-console-plugin-cert" not found Jul 07 19:01:32.845837 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:32.845812 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vrwmc_61a2f3ec-3a02-49c4-984b-ca722d4f7914/node-ca/0.log" Jul 07 19:01:33.077910 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.077832 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-p5nqd"] Jul 07 19:01:33.079947 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.079928 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-p5nqd" Jul 07 19:01:33.082742 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.082721 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 07 19:01:33.083002 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.082987 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 07 19:01:33.083307 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.083288 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 07 19:01:33.083437 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.083417 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 07 19:01:33.083575 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.083557 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-2nh6j\"" Jul 07 19:01:33.102336 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.102315 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-p5nqd"] Jul 07 19:01:33.108517 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.108495 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkb75\" (UniqueName: \"kubernetes.io/projected/777e2940-32bb-437a-ae18-8aac4a460fba-kube-api-access-gkb75\") pod \"service-ca-758f599c5-p5nqd\" (UID: \"777e2940-32bb-437a-ae18-8aac4a460fba\") " pod="openshift-service-ca/service-ca-758f599c5-p5nqd" Jul 07 19:01:33.108619 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.108568 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/777e2940-32bb-437a-ae18-8aac4a460fba-signing-key\") pod \"service-ca-758f599c5-p5nqd\" (UID: \"777e2940-32bb-437a-ae18-8aac4a460fba\") " pod="openshift-service-ca/service-ca-758f599c5-p5nqd" Jul 07 19:01:33.108619 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.108607 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/777e2940-32bb-437a-ae18-8aac4a460fba-signing-cabundle\") pod \"service-ca-758f599c5-p5nqd\" (UID: \"777e2940-32bb-437a-ae18-8aac4a460fba\") " pod="openshift-service-ca/service-ca-758f599c5-p5nqd" Jul 07 19:01:33.209331 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.209277 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkb75\" (UniqueName: \"kubernetes.io/projected/777e2940-32bb-437a-ae18-8aac4a460fba-kube-api-access-gkb75\") pod \"service-ca-758f599c5-p5nqd\" (UID: \"777e2940-32bb-437a-ae18-8aac4a460fba\") " pod="openshift-service-ca/service-ca-758f599c5-p5nqd" Jul 07 19:01:33.209530 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.209501 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/777e2940-32bb-437a-ae18-8aac4a460fba-signing-key\") pod \"service-ca-758f599c5-p5nqd\" (UID: \"777e2940-32bb-437a-ae18-8aac4a460fba\") " pod="openshift-service-ca/service-ca-758f599c5-p5nqd" Jul 07 19:01:33.209589 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.209554 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/777e2940-32bb-437a-ae18-8aac4a460fba-signing-cabundle\") pod \"service-ca-758f599c5-p5nqd\" (UID: \"777e2940-32bb-437a-ae18-8aac4a460fba\") " pod="openshift-service-ca/service-ca-758f599c5-p5nqd" Jul 07 19:01:33.210280 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.210259 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/777e2940-32bb-437a-ae18-8aac4a460fba-signing-cabundle\") pod \"service-ca-758f599c5-p5nqd\" (UID: \"777e2940-32bb-437a-ae18-8aac4a460fba\") " pod="openshift-service-ca/service-ca-758f599c5-p5nqd" Jul 07 19:01:33.211838 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.211819 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/777e2940-32bb-437a-ae18-8aac4a460fba-signing-key\") pod \"service-ca-758f599c5-p5nqd\" (UID: \"777e2940-32bb-437a-ae18-8aac4a460fba\") " pod="openshift-service-ca/service-ca-758f599c5-p5nqd" Jul 07 19:01:33.222288 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.222265 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkb75\" (UniqueName: \"kubernetes.io/projected/777e2940-32bb-437a-ae18-8aac4a460fba-kube-api-access-gkb75\") pod \"service-ca-758f599c5-p5nqd\" (UID: \"777e2940-32bb-437a-ae18-8aac4a460fba\") " pod="openshift-service-ca/service-ca-758f599c5-p5nqd" Jul 07 19:01:33.388398 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.388361 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-p5nqd" Jul 07 19:01:33.411885 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.411855 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:33.412020 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.411951 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:33.412020 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.411978 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:33.412155 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:33.412032 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:01:33.412155 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:33.412089 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 19:01:33.412155 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:33.412101 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69d7bf5b4f-gns76: secret "image-registry-tls" not found Jul 07 19:01:33.412155 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:33.412099 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 19:01:33.412155 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:33.412131 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls podName:d200611b-531a-49e6-8d44-20ba3ac01a4a nodeName:}" failed. No retries permitted until 2026-07-07 19:01:37.412106842 +0000 UTC m=+67.604646908 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-trklp" (UID: "d200611b-531a-49e6-8d44-20ba3ac01a4a") : secret "insights-runtime-extractor-tls" not found Jul 07 19:01:33.412155 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:33.412151 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls podName:10c78cf7-3156-415b-8278-bf57dd5290b0 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:41.412141187 +0000 UTC m=+71.604681238 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls") pod "image-registry-69d7bf5b4f-gns76" (UID: "10c78cf7-3156-415b-8278-bf57dd5290b0") : secret "image-registry-tls" not found Jul 07 19:01:33.412361 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:33.412165 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-rbac-proxy-cm podName:d200611b-531a-49e6-8d44-20ba3ac01a4a nodeName:}" failed. No retries permitted until 2026-07-07 19:01:37.412157493 +0000 UTC m=+67.604697560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-trklp" (UID: "d200611b-531a-49e6-8d44-20ba3ac01a4a") : configmap "kube-rbac-proxy" not found Jul 07 19:01:33.510954 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.510925 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-p5nqd"] Jul 07 19:01:33.514675 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:01:33.514648 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777e2940_32bb_437a_ae18_8aac4a460fba.slice/crio-850527a24581d78edca69d6c2ec0d55ca7c03aef6eaa7d4f83114934cd03dd93 WatchSource:0}: Error finding container 850527a24581d78edca69d6c2ec0d55ca7c03aef6eaa7d4f83114934cd03dd93: Status 404 returned error can't find the container with id 850527a24581d78edca69d6c2ec0d55ca7c03aef6eaa7d4f83114934cd03dd93 Jul 07 19:01:33.647976 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.647883 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-p5nqd" event={"ID":"777e2940-32bb-437a-ae18-8aac4a460fba","Type":"ContainerStarted","Data":"850527a24581d78edca69d6c2ec0d55ca7c03aef6eaa7d4f83114934cd03dd93"} Jul 07 19:01:33.648728 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.648709 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vz9jl_08cae73a-9871-4cf5-a4d5-1045cc284553/serve-healthcheck-canary/0.log" Jul 07 19:01:33.816378 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:33.816339 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6487b\" (UID: \"cadb609a-f2e9-4ca0-a4eb-85952ff8160c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6487b" Jul 07 19:01:33.816743 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:33.816459 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 19:01:33.816743 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:33.816512 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert podName:cadb609a-f2e9-4ca0-a4eb-85952ff8160c nodeName:}" failed. No retries permitted until 2026-07-07 19:01:35.81649877 +0000 UTC m=+66.009038814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-6487b" (UID: "cadb609a-f2e9-4ca0-a4eb-85952ff8160c") : secret "networking-console-plugin-cert" not found Jul 07 19:01:35.025917 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:35.025875 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret\") pod \"global-pull-secret-syncer-n6cxp\" (UID: \"c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac\") " pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:01:35.025917 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:35.025920 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs\") pod \"network-metrics-daemon-rfz5t\" (UID: \"848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c\") " pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:01:35.028361 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:35.028341 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 19:01:35.028438 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:35.028351 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 19:01:35.036367 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:35.036342 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 19:01:35.036468 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:35.036414 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs podName:848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c nodeName:}" failed. No retries permitted until 2026-07-07 19:02:39.036393235 +0000 UTC m=+129.228933300 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs") pod "network-metrics-daemon-rfz5t" (UID: "848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c") : secret "metrics-daemon-secret" not found Jul 07 19:01:35.038513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:35.038480 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac-original-pull-secret\") pod \"global-pull-secret-syncer-n6cxp\" (UID: \"c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac\") " pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:01:35.228117 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:35.228073 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8g6k\" (UniqueName: \"kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k\") pod \"network-check-target-8m6sh\" (UID: \"1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b\") " pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:01:35.230518 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:35.230492 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 19:01:35.240905 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:35.240880 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 19:01:35.243158 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:35.243137 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n6cxp" Jul 07 19:01:35.251546 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:35.251520 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8g6k\" (UniqueName: \"kubernetes.io/projected/1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b-kube-api-access-c8g6k\") pod \"network-check-target-8m6sh\" (UID: \"1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b\") " pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:01:35.375297 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:35.375267 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n6cxp"] Jul 07 19:01:35.378528 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:01:35.378484 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14f74dc_d3d9_4962_9dbc_7a3209b3d4ac.slice/crio-e5890122c560dbcd98fceaa5ff12ebf1ba016aaa5cf532a255f244a9127ba61c WatchSource:0}: Error finding container e5890122c560dbcd98fceaa5ff12ebf1ba016aaa5cf532a255f244a9127ba61c: Status 404 returned error can't find the container with id e5890122c560dbcd98fceaa5ff12ebf1ba016aaa5cf532a255f244a9127ba61c Jul 07 19:01:35.552715 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:35.552637 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6dffl\"" Jul 07 19:01:35.559122 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:35.559105 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:01:35.654721 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:35.654685 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n6cxp" event={"ID":"c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac","Type":"ContainerStarted","Data":"e5890122c560dbcd98fceaa5ff12ebf1ba016aaa5cf532a255f244a9127ba61c"} Jul 07 19:01:35.699105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:35.699073 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8m6sh"] Jul 07 19:01:35.702784 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:01:35.702758 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cfc45a0_4bb3_4c06_b2c0_a75812dc6f8b.slice/crio-f90c5895af95de915edd2ad1fbf453494a00636e8207f676e990197aecf873c8 WatchSource:0}: Error finding container f90c5895af95de915edd2ad1fbf453494a00636e8207f676e990197aecf873c8: Status 404 returned error can't find the container with id f90c5895af95de915edd2ad1fbf453494a00636e8207f676e990197aecf873c8 Jul 07 19:01:35.833746 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:35.833650 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6487b\" (UID: \"cadb609a-f2e9-4ca0-a4eb-85952ff8160c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6487b" Jul 07 19:01:35.833903 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:35.833806 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 19:01:35.833903 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:35.833867 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert podName:cadb609a-f2e9-4ca0-a4eb-85952ff8160c nodeName:}" failed. No retries permitted until 2026-07-07 19:01:39.833853511 +0000 UTC m=+70.026393559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-6487b" (UID: "cadb609a-f2e9-4ca0-a4eb-85952ff8160c") : secret "networking-console-plugin-cert" not found Jul 07 19:01:36.658833 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:36.658782 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8m6sh" event={"ID":"1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b","Type":"ContainerStarted","Data":"f90c5895af95de915edd2ad1fbf453494a00636e8207f676e990197aecf873c8"} Jul 07 19:01:37.446890 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:37.446850 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:37.447102 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:37.446907 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:37.447102 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:37.446989 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 19:01:37.447102 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:37.447007 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:01:37.447102 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:37.447069 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls podName:d200611b-531a-49e6-8d44-20ba3ac01a4a nodeName:}" failed. No retries permitted until 2026-07-07 19:01:45.447056536 +0000 UTC m=+75.639596594 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-trklp" (UID: "d200611b-531a-49e6-8d44-20ba3ac01a4a") : secret "insights-runtime-extractor-tls" not found Jul 07 19:01:37.447102 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:37.447083 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-rbac-proxy-cm podName:d200611b-531a-49e6-8d44-20ba3ac01a4a nodeName:}" failed. No retries permitted until 2026-07-07 19:01:45.447076763 +0000 UTC m=+75.639616808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-trklp" (UID: "d200611b-531a-49e6-8d44-20ba3ac01a4a") : configmap "kube-rbac-proxy" not found Jul 07 19:01:37.664892 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:37.664842 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-p5nqd" event={"ID":"777e2940-32bb-437a-ae18-8aac4a460fba","Type":"ContainerStarted","Data":"55cf01302b9c8b84e13878eae6703b776e5f003d8e0d4b5f1441fbc1d4d43571"} Jul 07 19:01:37.691393 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:37.691312 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-p5nqd" podStartSLOduration=1.197346442 podStartE2EDuration="4.691292584s" totalImagesPullingTime="3.493946142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:37.689579698 +0000 UTC m=+67.882119767" watchObservedRunningTime="2026-07-07 19:01:37.691292584 +0000 UTC m=+67.883832652" Jul 07 19:01:38.050370 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:38.050335 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls\") pod \"dns-default-hrh48\" (UID: \"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74\") " pod="openshift-dns/dns-default-hrh48" Jul 07 19:01:38.050575 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:38.050506 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:01:38.050645 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:38.050576 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls podName:34b59a25-fd9d-4ed1-befe-7a9d4ad5db74 nodeName:}" failed. No retries permitted until 2026-07-07 19:02:10.050556068 +0000 UTC m=+100.243096139 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls") pod "dns-default-hrh48" (UID: "34b59a25-fd9d-4ed1-befe-7a9d4ad5db74") : secret "dns-default-metrics-tls" not found Jul 07 19:01:39.865812 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:39.865764 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6487b\" (UID: \"cadb609a-f2e9-4ca0-a4eb-85952ff8160c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6487b" Jul 07 19:01:39.866311 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:39.865909 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 19:01:39.866311 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:39.865987 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert podName:cadb609a-f2e9-4ca0-a4eb-85952ff8160c nodeName:}" failed. No retries permitted until 2026-07-07 19:01:47.865968015 +0000 UTC m=+78.058508063 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-6487b" (UID: "cadb609a-f2e9-4ca0-a4eb-85952ff8160c") : secret "networking-console-plugin-cert" not found Jul 07 19:01:41.481802 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:41.481763 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls\") pod \"image-registry-69d7bf5b4f-gns76\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:41.482319 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:41.481920 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 07 19:01:41.482319 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:41.481940 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69d7bf5b4f-gns76: secret "image-registry-tls" not found Jul 07 19:01:41.482319 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:41.481992 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls podName:10c78cf7-3156-415b-8278-bf57dd5290b0 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:57.481978019 +0000 UTC m=+87.674518065 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls") pod "image-registry-69d7bf5b4f-gns76" (UID: "10c78cf7-3156-415b-8278-bf57dd5290b0") : secret "image-registry-tls" not found Jul 07 19:01:41.682452 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:41.682393 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8m6sh" event={"ID":"1cfc45a0-4bb3-4c06-b2c0-a75812dc6f8b","Type":"ContainerStarted","Data":"b5fc60fca50e0e8eb6eb9df9b001a7fef1ae0ed4de3724c140f2e7a347a4e05c"} Jul 07 19:01:41.682616 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:41.682506 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:01:41.684147 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:41.684118 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n6cxp" event={"ID":"c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac","Type":"ContainerStarted","Data":"f2cdecabecdb0270dd1894b3d7c7bf2e961fbdaae398d978c862b9daa3ee4be9"} Jul 07 19:01:41.701161 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:41.701099 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8m6sh" podStartSLOduration=65.302168282 podStartE2EDuration="1m10.70108262s" totalImagesPullingTime="5.398914338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:41.700790251 +0000 UTC m=+71.893330320" watchObservedRunningTime="2026-07-07 19:01:41.70108262 +0000 UTC m=+71.893622690" Jul 07 19:01:41.719552 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:41.719488 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-n6cxp" podStartSLOduration=64.995815879 podStartE2EDuration="1m10.719472192s" totalImagesPullingTime="5.723656313s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:41.718926787 +0000 UTC m=+71.911466854" watchObservedRunningTime="2026-07-07 19:01:41.719472192 +0000 UTC m=+71.912012289" Jul 07 19:01:45.516188 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:45.516150 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:45.516781 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:45.516213 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:45.516781 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:45.516349 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:01:45.516781 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:45.516413 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls podName:d200611b-531a-49e6-8d44-20ba3ac01a4a nodeName:}" failed. No retries permitted until 2026-07-07 19:02:01.516400448 +0000 UTC m=+91.708940493 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-trklp" (UID: "d200611b-531a-49e6-8d44-20ba3ac01a4a") : secret "insights-runtime-extractor-tls" not found Jul 07 19:01:45.516956 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:45.516845 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d200611b-531a-49e6-8d44-20ba3ac01a4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:01:47.937367 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:47.937324 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6487b\" (UID: \"cadb609a-f2e9-4ca0-a4eb-85952ff8160c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6487b" Jul 07 19:01:47.937862 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:47.937513 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 07 19:01:47.937862 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:47.937600 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert podName:cadb609a-f2e9-4ca0-a4eb-85952ff8160c nodeName:}" failed. No retries permitted until 2026-07-07 19:02:03.937579709 +0000 UTC m=+94.130119763 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-6487b" (UID: "cadb609a-f2e9-4ca0-a4eb-85952ff8160c") : secret "networking-console-plugin-cert" not found Jul 07 19:01:54.490368 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.490249 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cc78b9d8-pfd28"] Jul 07 19:01:54.493248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.493231 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cc78b9d8-pfd28" Jul 07 19:01:54.505115 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.505093 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 07 19:01:54.506453 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.506432 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 07 19:01:54.507121 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.507102 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 07 19:01:54.507214 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.507191 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-lf77m\"" Jul 07 19:01:54.507272 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.507216 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 07 19:01:54.513343 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.513321 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cc78b9d8-pfd28"] Jul 07 19:01:54.551983 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.551951 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69d7bf5b4f-gns76"] Jul 07 19:01:54.552177 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:01:54.552157 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" podUID="10c78cf7-3156-415b-8278-bf57dd5290b0" Jul 07 19:01:54.592776 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.592745 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6md4h\" (UniqueName: \"kubernetes.io/projected/1ecf0890-2545-45a3-9794-0bda9023fdd1-kube-api-access-6md4h\") pod \"managed-serviceaccount-addon-agent-86cc78b9d8-pfd28\" (UID: \"1ecf0890-2545-45a3-9794-0bda9023fdd1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cc78b9d8-pfd28" Jul 07 19:01:54.592923 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.592834 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ecf0890-2545-45a3-9794-0bda9023fdd1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86cc78b9d8-pfd28\" (UID: \"1ecf0890-2545-45a3-9794-0bda9023fdd1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cc78b9d8-pfd28" Jul 07 19:01:54.666103 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.666073 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76f5b666df-rl2xc"] Jul 07 19:01:54.668101 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.668086 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.693748 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.693714 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6md4h\" (UniqueName: \"kubernetes.io/projected/1ecf0890-2545-45a3-9794-0bda9023fdd1-kube-api-access-6md4h\") pod \"managed-serviceaccount-addon-agent-86cc78b9d8-pfd28\" (UID: \"1ecf0890-2545-45a3-9794-0bda9023fdd1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cc78b9d8-pfd28" Jul 07 19:01:54.693893 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.693874 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ecf0890-2545-45a3-9794-0bda9023fdd1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86cc78b9d8-pfd28\" (UID: \"1ecf0890-2545-45a3-9794-0bda9023fdd1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cc78b9d8-pfd28" Jul 07 19:01:54.696283 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.696262 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1ecf0890-2545-45a3-9794-0bda9023fdd1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86cc78b9d8-pfd28\" (UID: \"1ecf0890-2545-45a3-9794-0bda9023fdd1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cc78b9d8-pfd28" Jul 07 19:01:54.707601 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.707572 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6md4h\" (UniqueName: \"kubernetes.io/projected/1ecf0890-2545-45a3-9794-0bda9023fdd1-kube-api-access-6md4h\") pod \"managed-serviceaccount-addon-agent-86cc78b9d8-pfd28\" (UID: \"1ecf0890-2545-45a3-9794-0bda9023fdd1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cc78b9d8-pfd28" Jul 07 19:01:54.724668 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.724643 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:54.730005 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.729985 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:54.763933 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.763863 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76f5b666df-rl2xc"] Jul 07 19:01:54.794332 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.794309 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10c78cf7-3156-415b-8278-bf57dd5290b0-trusted-ca\") pod \"10c78cf7-3156-415b-8278-bf57dd5290b0\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " Jul 07 19:01:54.794475 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.794341 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-bound-sa-token\") pod \"10c78cf7-3156-415b-8278-bf57dd5290b0\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " Jul 07 19:01:54.794475 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.794362 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10c78cf7-3156-415b-8278-bf57dd5290b0-installation-pull-secrets\") pod \"10c78cf7-3156-415b-8278-bf57dd5290b0\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " Jul 07 19:01:54.794475 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.794387 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10c78cf7-3156-415b-8278-bf57dd5290b0-ca-trust-extracted\") pod \"10c78cf7-3156-415b-8278-bf57dd5290b0\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " Jul 07 19:01:54.794475 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.794429 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-certificates\") pod \"10c78cf7-3156-415b-8278-bf57dd5290b0\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " Jul 07 19:01:54.794475 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.794458 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10c78cf7-3156-415b-8278-bf57dd5290b0-image-registry-private-configuration\") pod \"10c78cf7-3156-415b-8278-bf57dd5290b0\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " Jul 07 19:01:54.794721 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.794497 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9d8m\" (UniqueName: \"kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-kube-api-access-h9d8m\") pod \"10c78cf7-3156-415b-8278-bf57dd5290b0\" (UID: \"10c78cf7-3156-415b-8278-bf57dd5290b0\") " Jul 07 19:01:54.794721 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.794653 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krh4r\" (UniqueName: \"kubernetes.io/projected/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-kube-api-access-krh4r\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.794834 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.794728 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-image-registry-private-configuration\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.794834 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.794755 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-registry-tls\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.794834 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.794733 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c78cf7-3156-415b-8278-bf57dd5290b0-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "10c78cf7-3156-415b-8278-bf57dd5290b0" (UID: "10c78cf7-3156-415b-8278-bf57dd5290b0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:01:54.794993 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.794900 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-trusted-ca\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.794993 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.794950 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c78cf7-3156-415b-8278-bf57dd5290b0-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "10c78cf7-3156-415b-8278-bf57dd5290b0" (UID: "10c78cf7-3156-415b-8278-bf57dd5290b0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:01:54.794993 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.794976 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-installation-pull-secrets\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.795183 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.794990 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "10c78cf7-3156-415b-8278-bf57dd5290b0" (UID: "10c78cf7-3156-415b-8278-bf57dd5290b0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:01:54.795183 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.795004 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-bound-sa-token\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.795183 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.795112 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-ca-trust-extracted\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.795183 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.795141 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-registry-certificates\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.795183 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.795182 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10c78cf7-3156-415b-8278-bf57dd5290b0-trusted-ca\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:01:54.795428 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.795197 2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/10c78cf7-3156-415b-8278-bf57dd5290b0-ca-trust-extracted\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:01:54.795428 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.795214 2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-certificates\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:01:54.796778 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.796751 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c78cf7-3156-415b-8278-bf57dd5290b0-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "10c78cf7-3156-415b-8278-bf57dd5290b0" (UID: "10c78cf7-3156-415b-8278-bf57dd5290b0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:01:54.796911 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.796862 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c78cf7-3156-415b-8278-bf57dd5290b0-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "10c78cf7-3156-415b-8278-bf57dd5290b0" (UID: "10c78cf7-3156-415b-8278-bf57dd5290b0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:01:54.796911 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.796875 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "10c78cf7-3156-415b-8278-bf57dd5290b0" (UID: "10c78cf7-3156-415b-8278-bf57dd5290b0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:01:54.797022 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.796954 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-kube-api-access-h9d8m" (OuterVolumeSpecName: "kube-api-access-h9d8m") pod "10c78cf7-3156-415b-8278-bf57dd5290b0" (UID: "10c78cf7-3156-415b-8278-bf57dd5290b0"). InnerVolumeSpecName "kube-api-access-h9d8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:01:54.818725 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.818700 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cc78b9d8-pfd28" Jul 07 19:01:54.896752 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.896417 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-registry-certificates\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.896752 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.896512 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krh4r\" (UniqueName: \"kubernetes.io/projected/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-kube-api-access-krh4r\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.896752 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.896576 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-image-registry-private-configuration\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.896752 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.896601 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-registry-tls\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.896752 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.896640 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-trusted-ca\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.896752 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.896680 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-installation-pull-secrets\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.896752 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.896704 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-bound-sa-token\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.896752 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.896748 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-ca-trust-extracted\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.897485 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.896798 2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-bound-sa-token\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:01:54.897485 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.897145 2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/10c78cf7-3156-415b-8278-bf57dd5290b0-installation-pull-secrets\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:01:54.897485 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.897166 2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/10c78cf7-3156-415b-8278-bf57dd5290b0-image-registry-private-configuration\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:01:54.897485 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.897181 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9d8m\" (UniqueName: \"kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-kube-api-access-h9d8m\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:01:54.897685 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.897642 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-ca-trust-extracted\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.898088 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.898063 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-registry-certificates\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.898588 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.898567 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-trusted-ca\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.899938 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.899895 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-installation-pull-secrets\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.899938 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.899913 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-image-registry-private-configuration\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.900329 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.900314 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-registry-tls\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.919739 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.919705 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-bound-sa-token\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.922812 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.922789 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krh4r\" (UniqueName: \"kubernetes.io/projected/9a2baf66-4d94-4bca-b8c8-7d64df15ad0e-kube-api-access-krh4r\") pod \"image-registry-76f5b666df-rl2xc\" (UID: \"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e\") " pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:54.960080 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.960036 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cc78b9d8-pfd28"] Jul 07 19:01:54.963162 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:01:54.963134 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ecf0890_2545_45a3_9794_0bda9023fdd1.slice/crio-4da5c2829f12f8fd6357770c48f2f88e596edc3321150bbef29365990731df92 WatchSource:0}: Error finding container 4da5c2829f12f8fd6357770c48f2f88e596edc3321150bbef29365990731df92: Status 404 returned error can't find the container with id 4da5c2829f12f8fd6357770c48f2f88e596edc3321150bbef29365990731df92 Jul 07 19:01:54.979082 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.979056 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8tx7z\"" Jul 07 19:01:54.987731 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:54.987705 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:55.141547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:55.141508 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76f5b666df-rl2xc"] Jul 07 19:01:55.144188 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:01:55.144167 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2baf66_4d94_4bca_b8c8_7d64df15ad0e.slice/crio-47fe78db13d3ec2eb89eed704a3e0e857a3ccbd52b074bbefd31d7bbe1e6a56a WatchSource:0}: Error finding container 47fe78db13d3ec2eb89eed704a3e0e857a3ccbd52b074bbefd31d7bbe1e6a56a: Status 404 returned error can't find the container with id 47fe78db13d3ec2eb89eed704a3e0e857a3ccbd52b074bbefd31d7bbe1e6a56a Jul 07 19:01:55.733891 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:55.733848 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" event={"ID":"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e","Type":"ContainerStarted","Data":"bb675214a3a522482d33a64c41138f893e7527f14a79cc9f97d428524b5c4559"} Jul 07 19:01:55.734399 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:55.733946 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" event={"ID":"9a2baf66-4d94-4bca-b8c8-7d64df15ad0e","Type":"ContainerStarted","Data":"47fe78db13d3ec2eb89eed704a3e0e857a3ccbd52b074bbefd31d7bbe1e6a56a"} Jul 07 19:01:55.734924 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:55.734900 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:01:55.736427 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:55.736384 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cc78b9d8-pfd28" event={"ID":"1ecf0890-2545-45a3-9794-0bda9023fdd1","Type":"ContainerStarted","Data":"4da5c2829f12f8fd6357770c48f2f88e596edc3321150bbef29365990731df92"} Jul 07 19:01:55.736427 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:55.736419 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69d7bf5b4f-gns76" Jul 07 19:01:55.766905 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:55.766827 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" podStartSLOduration=1.766808711 podStartE2EDuration="1.766808711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:55.766473014 +0000 UTC m=+85.959013085" watchObservedRunningTime="2026-07-07 19:01:55.766808711 +0000 UTC m=+85.959348780" Jul 07 19:01:55.813581 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:55.813545 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69d7bf5b4f-gns76"] Jul 07 19:01:55.822643 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:55.822607 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69d7bf5b4f-gns76"] Jul 07 19:01:55.908175 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:55.908135 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/10c78cf7-3156-415b-8278-bf57dd5290b0-registry-tls\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:01:56.327337 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:56.327299 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10c78cf7-3156-415b-8278-bf57dd5290b0" path="/var/lib/kubelet/pods/10c78cf7-3156-415b-8278-bf57dd5290b0/volumes" Jul 07 19:01:57.744887 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:57.744854 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cc78b9d8-pfd28" event={"ID":"1ecf0890-2545-45a3-9794-0bda9023fdd1","Type":"ContainerStarted","Data":"66f9c4b1ca259820d7b4b49be6195c06db5178ce960be0cf17eef1bf5dc6a9bd"} Jul 07 19:01:57.768184 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:01:57.768140 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86cc78b9d8-pfd28" podStartSLOduration=1.554525628 podStartE2EDuration="3.768122432s" totalImagesPullingTime="2.213596804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:57.767745836 +0000 UTC m=+87.960285881" watchObservedRunningTime="2026-07-07 19:01:57.768122432 +0000 UTC m=+87.960662530" Jul 07 19:02:01.320543 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:01.320502 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-d9lht"] Jul 07 19:02:01.354614 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:01.354572 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-d9lht"] Jul 07 19:02:01.354778 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:01.354689 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-d9lht" Jul 07 19:02:01.358179 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:01.358155 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gbf77\"" Jul 07 19:02:01.359517 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:01.359495 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 07 19:02:01.455780 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:01.455735 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2bd2a75e-64e6-4544-b6ce-b705f34214f3-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-d9lht\" (UID: \"2bd2a75e-64e6-4544-b6ce-b705f34214f3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-d9lht" Jul 07 19:02:01.557193 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:01.557154 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2bd2a75e-64e6-4544-b6ce-b705f34214f3-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-d9lht\" (UID: \"2bd2a75e-64e6-4544-b6ce-b705f34214f3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-d9lht" Jul 07 19:02:01.557336 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:01.557208 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:02:01.559445 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:01.559415 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d200611b-531a-49e6-8d44-20ba3ac01a4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-trklp\" (UID: \"d200611b-531a-49e6-8d44-20ba3ac01a4a\") " pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:02:01.559652 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:01.559634 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2bd2a75e-64e6-4544-b6ce-b705f34214f3-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-d9lht\" (UID: \"2bd2a75e-64e6-4544-b6ce-b705f34214f3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-d9lht" Jul 07 19:02:01.663094 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:01.663031 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-d9lht" Jul 07 19:02:01.820285 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:01.820255 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-d9lht"] Jul 07 19:02:01.825389 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:02:01.825348 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd2a75e_64e6_4544_b6ce_b705f34214f3.slice/crio-5025ba9e3b4e5d112523b56ce231989fea838d0e51cdeb55924c983c38486989 WatchSource:0}: Error finding container 5025ba9e3b4e5d112523b56ce231989fea838d0e51cdeb55924c983c38486989: Status 404 returned error can't find the container with id 5025ba9e3b4e5d112523b56ce231989fea838d0e51cdeb55924c983c38486989 Jul 07 19:02:01.834729 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:01.834706 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bj6vq\"" Jul 07 19:02:01.842741 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:01.842719 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-trklp" Jul 07 19:02:02.007927 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:02.007894 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-trklp"] Jul 07 19:02:02.010877 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:02:02.010841 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd200611b_531a_49e6_8d44_20ba3ac01a4a.slice/crio-0323297e4c2bad66df7f8f6f49f3427fdb7f8f2eed009dfce34df748802a2a43 WatchSource:0}: Error finding container 0323297e4c2bad66df7f8f6f49f3427fdb7f8f2eed009dfce34df748802a2a43: Status 404 returned error can't find the container with id 0323297e4c2bad66df7f8f6f49f3427fdb7f8f2eed009dfce34df748802a2a43 Jul 07 19:02:02.763458 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:02.763417 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-trklp" event={"ID":"d200611b-531a-49e6-8d44-20ba3ac01a4a","Type":"ContainerStarted","Data":"0323297e4c2bad66df7f8f6f49f3427fdb7f8f2eed009dfce34df748802a2a43"} Jul 07 19:02:02.766397 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:02.766358 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-d9lht" event={"ID":"2bd2a75e-64e6-4544-b6ce-b705f34214f3","Type":"ContainerStarted","Data":"5025ba9e3b4e5d112523b56ce231989fea838d0e51cdeb55924c983c38486989"} Jul 07 19:02:03.771277 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:03.771196 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-d9lht" event={"ID":"2bd2a75e-64e6-4544-b6ce-b705f34214f3","Type":"ContainerStarted","Data":"ffbcd009a648ff2acac14fd5caa7cd2082d26ed75ce3eeb3bf02f8fe4d255d65"} Jul 07 19:02:03.771837 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:03.771403 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-d9lht" Jul 07 19:02:03.777077 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:03.776805 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-d9lht" Jul 07 19:02:03.790929 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:03.790884 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-d9lht" podStartSLOduration=1.185859011 podStartE2EDuration="2.790867373s" totalImagesPullingTime="1.605008362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:02:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:03.790272069 +0000 UTC m=+93.982812137" watchObservedRunningTime="2026-07-07 19:02:03.790867373 +0000 UTC m=+93.983407441" Jul 07 19:02:03.981744 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:03.981691 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6487b\" (UID: \"cadb609a-f2e9-4ca0-a4eb-85952ff8160c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6487b" Jul 07 19:02:03.984358 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:03.984327 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cadb609a-f2e9-4ca0-a4eb-85952ff8160c-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-6487b\" (UID: \"cadb609a-f2e9-4ca0-a4eb-85952ff8160c\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-6487b" Jul 07 19:02:03.985112 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:03.985088 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6487b" Jul 07 19:02:06.190503 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:06.190472 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-6487b"] Jul 07 19:02:06.193596 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:02:06.193553 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcadb609a_f2e9_4ca0_a4eb_85952ff8160c.slice/crio-0d315094f40c8d48d09feddc5a1f00dcf34a0e8c6c6502f9fa94a45f941b094c WatchSource:0}: Error finding container 0d315094f40c8d48d09feddc5a1f00dcf34a0e8c6c6502f9fa94a45f941b094c: Status 404 returned error can't find the container with id 0d315094f40c8d48d09feddc5a1f00dcf34a0e8c6c6502f9fa94a45f941b094c Jul 07 19:02:06.783995 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:06.783848 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-trklp" event={"ID":"d200611b-531a-49e6-8d44-20ba3ac01a4a","Type":"ContainerStarted","Data":"4ed29dc5be3f7782043d1367652de758b04817c8e06cb432dea4f937bf0744b0"} Jul 07 19:02:06.784903 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:06.784878 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6487b" event={"ID":"cadb609a-f2e9-4ca0-a4eb-85952ff8160c","Type":"ContainerStarted","Data":"0d315094f40c8d48d09feddc5a1f00dcf34a0e8c6c6502f9fa94a45f941b094c"} Jul 07 19:02:07.791032 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:07.790932 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-trklp" event={"ID":"d200611b-531a-49e6-8d44-20ba3ac01a4a","Type":"ContainerStarted","Data":"e12205b5da441e6013f358e39106df55fd8ce4c2ca7b4924b685ade951c5643b"} Jul 07 19:02:07.792662 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:07.792623 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6487b" event={"ID":"cadb609a-f2e9-4ca0-a4eb-85952ff8160c","Type":"ContainerStarted","Data":"52526bc64e86ea2b7866144e609e63db25074073bf267a95b124946430ee1675"} Jul 07 19:02:07.813331 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:07.813284 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-6487b" podStartSLOduration=34.53368095 podStartE2EDuration="35.8132663s" totalImagesPullingTime="1.27958535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:07.812431352 +0000 UTC m=+98.004971419" watchObservedRunningTime="2026-07-07 19:02:07.8132663 +0000 UTC m=+98.005806374" Jul 07 19:02:09.801441 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:09.801404 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-trklp" event={"ID":"d200611b-531a-49e6-8d44-20ba3ac01a4a","Type":"ContainerStarted","Data":"830423d73bb56cf8a0bf87a1ba068585c6ae9484894df398e490b244b67c9ddc"} Jul 07 19:02:09.823329 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:09.823272 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-trklp" podStartSLOduration=33.269091985 podStartE2EDuration="40.823255955s" totalImagesPullingTime="7.55416397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:01:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:09.822431341 +0000 UTC m=+100.014971408" watchObservedRunningTime="2026-07-07 19:02:09.823255955 +0000 UTC m=+100.015796018" Jul 07 19:02:10.136539 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:10.136506 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls\") pod \"dns-default-hrh48\" (UID: \"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74\") " pod="openshift-dns/dns-default-hrh48" Jul 07 19:02:10.138811 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:10.138786 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/34b59a25-fd9d-4ed1-befe-7a9d4ad5db74-metrics-tls\") pod \"dns-default-hrh48\" (UID: \"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74\") " pod="openshift-dns/dns-default-hrh48" Jul 07 19:02:10.382467 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:10.382428 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-d4n46\"" Jul 07 19:02:10.390815 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:10.390736 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hrh48" Jul 07 19:02:10.519564 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:10.519533 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hrh48"] Jul 07 19:02:10.523769 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:02:10.523745 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b59a25_fd9d_4ed1_befe_7a9d4ad5db74.slice/crio-93ad0a14fd79db4be71fe9f944b39c1961c0fdb6bbe68ceec47e7b4c6ee9bd7a WatchSource:0}: Error finding container 93ad0a14fd79db4be71fe9f944b39c1961c0fdb6bbe68ceec47e7b4c6ee9bd7a: Status 404 returned error can't find the container with id 93ad0a14fd79db4be71fe9f944b39c1961c0fdb6bbe68ceec47e7b4c6ee9bd7a Jul 07 19:02:10.805526 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:10.805438 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hrh48" event={"ID":"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74","Type":"ContainerStarted","Data":"93ad0a14fd79db4be71fe9f944b39c1961c0fdb6bbe68ceec47e7b4c6ee9bd7a"} Jul 07 19:02:11.757991 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.757957 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv"] Jul 07 19:02:11.762536 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.762514 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv" Jul 07 19:02:11.764870 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.764848 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 07 19:02:11.765068 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.764860 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 07 19:02:11.765652 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.765627 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-nps5j\"" Jul 07 19:02:11.765766 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.765631 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 19:02:11.766194 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.766175 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 19:02:11.767743 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.767568 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 19:02:11.778515 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.778482 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv"] Jul 07 19:02:11.788685 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.787375 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm"] Jul 07 19:02:11.790967 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.790944 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:11.794309 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.794289 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 07 19:02:11.794425 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.794342 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-69mb7\"" Jul 07 19:02:11.794538 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.794522 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 07 19:02:11.794870 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.794854 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 07 19:02:11.806715 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.805752 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rvlhp"] Jul 07 19:02:11.811412 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.811164 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:11.814575 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.814552 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm"] Jul 07 19:02:11.816728 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.816707 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jc4zb\"" Jul 07 19:02:11.817353 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.817337 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 19:02:11.817587 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.817572 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 19:02:11.817849 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.817833 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 19:02:11.952152 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.952102 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-root\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:11.952325 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.952158 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/db8101fa-7197-464d-b9a8-385f5629f7ff-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-cdfrm\" (UID: \"db8101fa-7197-464d-b9a8-385f5629f7ff\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:11.952325 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.952236 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-node-exporter-wtmp\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:11.952464 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.952335 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db8101fa-7197-464d-b9a8-385f5629f7ff-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-cdfrm\" (UID: \"db8101fa-7197-464d-b9a8-385f5629f7ff\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:11.952464 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.952366 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0f243a4-a647-4eb4-bdbf-16638f205522-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-222dv\" (UID: \"c0f243a4-a647-4eb4-bdbf-16638f205522\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv" Jul 07 19:02:11.952464 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.952399 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/db8101fa-7197-464d-b9a8-385f5629f7ff-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-cdfrm\" (UID: \"db8101fa-7197-464d-b9a8-385f5629f7ff\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:11.952613 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.952471 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-sys\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:11.952613 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.952510 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:11.952613 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.952551 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0f243a4-a647-4eb4-bdbf-16638f205522-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-222dv\" (UID: \"c0f243a4-a647-4eb4-bdbf-16638f205522\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv" Jul 07 19:02:11.952613 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.952575 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-metrics-client-ca\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:11.952613 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.952606 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-node-exporter-tls\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:11.952857 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.952639 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:11.952857 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.952692 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t92zc\" (UniqueName: \"kubernetes.io/projected/db8101fa-7197-464d-b9a8-385f5629f7ff-kube-api-access-t92zc\") pod \"kube-state-metrics-7c9dbb8b7d-cdfrm\" (UID: \"db8101fa-7197-464d-b9a8-385f5629f7ff\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:11.952857 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.952726 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0f243a4-a647-4eb4-bdbf-16638f205522-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-222dv\" (UID: \"c0f243a4-a647-4eb4-bdbf-16638f205522\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv" Jul 07 19:02:11.952857 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.952750 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhqhl\" (UniqueName: \"kubernetes.io/projected/c0f243a4-a647-4eb4-bdbf-16638f205522-kube-api-access-rhqhl\") pod \"openshift-state-metrics-54cf96dfcf-222dv\" (UID: \"c0f243a4-a647-4eb4-bdbf-16638f205522\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv" Jul 07 19:02:11.952857 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.952778 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nnp6\" (UniqueName: \"kubernetes.io/projected/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-kube-api-access-6nnp6\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:11.952857 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.952815 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db8101fa-7197-464d-b9a8-385f5629f7ff-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-cdfrm\" (UID: \"db8101fa-7197-464d-b9a8-385f5629f7ff\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:11.952857 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.952854 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-node-exporter-textfile\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:11.953219 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:11.952978 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db8101fa-7197-464d-b9a8-385f5629f7ff-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-cdfrm\" (UID: \"db8101fa-7197-464d-b9a8-385f5629f7ff\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:12.053746 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.053647 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db8101fa-7197-464d-b9a8-385f5629f7ff-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-cdfrm\" (UID: \"db8101fa-7197-464d-b9a8-385f5629f7ff\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:12.053746 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.053700 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-root\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.053746 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.053728 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/db8101fa-7197-464d-b9a8-385f5629f7ff-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-cdfrm\" (UID: \"db8101fa-7197-464d-b9a8-385f5629f7ff\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:12.054019 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.053758 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-node-exporter-wtmp\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.054019 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.053815 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db8101fa-7197-464d-b9a8-385f5629f7ff-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-cdfrm\" (UID: \"db8101fa-7197-464d-b9a8-385f5629f7ff\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:12.054019 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.053836 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-root\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.054019 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.053846 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0f243a4-a647-4eb4-bdbf-16638f205522-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-222dv\" (UID: \"c0f243a4-a647-4eb4-bdbf-16638f205522\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv" Jul 07 19:02:12.054019 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.053887 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/db8101fa-7197-464d-b9a8-385f5629f7ff-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-cdfrm\" (UID: \"db8101fa-7197-464d-b9a8-385f5629f7ff\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:12.054019 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.053923 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-sys\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.054019 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.053939 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-node-exporter-wtmp\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.054019 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.053955 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.054019 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.054002 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0f243a4-a647-4eb4-bdbf-16638f205522-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-222dv\" (UID: \"c0f243a4-a647-4eb4-bdbf-16638f205522\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv" Jul 07 19:02:12.054592 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.054030 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-metrics-client-ca\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.054592 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.054089 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-node-exporter-tls\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.054592 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.054121 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.054592 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.054155 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92zc\" (UniqueName: \"kubernetes.io/projected/db8101fa-7197-464d-b9a8-385f5629f7ff-kube-api-access-t92zc\") pod \"kube-state-metrics-7c9dbb8b7d-cdfrm\" (UID: \"db8101fa-7197-464d-b9a8-385f5629f7ff\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:12.054592 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.054183 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0f243a4-a647-4eb4-bdbf-16638f205522-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-222dv\" (UID: \"c0f243a4-a647-4eb4-bdbf-16638f205522\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv" Jul 07 19:02:12.054592 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.054209 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqhl\" (UniqueName: \"kubernetes.io/projected/c0f243a4-a647-4eb4-bdbf-16638f205522-kube-api-access-rhqhl\") pod \"openshift-state-metrics-54cf96dfcf-222dv\" (UID: \"c0f243a4-a647-4eb4-bdbf-16638f205522\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv" Jul 07 19:02:12.054592 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.054233 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nnp6\" (UniqueName: \"kubernetes.io/projected/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-kube-api-access-6nnp6\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.054592 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.054261 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db8101fa-7197-464d-b9a8-385f5629f7ff-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-cdfrm\" (UID: \"db8101fa-7197-464d-b9a8-385f5629f7ff\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:12.054592 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.054285 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-node-exporter-textfile\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.054592 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.054439 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/db8101fa-7197-464d-b9a8-385f5629f7ff-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-cdfrm\" (UID: \"db8101fa-7197-464d-b9a8-385f5629f7ff\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:12.054592 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.054501 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-sys\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.054592 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.054563 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/db8101fa-7197-464d-b9a8-385f5629f7ff-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-cdfrm\" (UID: \"db8101fa-7197-464d-b9a8-385f5629f7ff\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:12.055183 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.054649 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-node-exporter-textfile\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.055183 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.054874 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.055183 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.054968 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-metrics-client-ca\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.055183 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:02:12.053816 2568 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 07 19:02:12.055183 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:02:12.055141 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db8101fa-7197-464d-b9a8-385f5629f7ff-kube-state-metrics-tls podName:db8101fa-7197-464d-b9a8-385f5629f7ff nodeName:}" failed. No retries permitted until 2026-07-07 19:02:12.555118046 +0000 UTC m=+102.747658098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/db8101fa-7197-464d-b9a8-385f5629f7ff-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-cdfrm" (UID: "db8101fa-7197-464d-b9a8-385f5629f7ff") : secret "kube-state-metrics-tls" not found Jul 07 19:02:12.055445 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:02:12.054665 2568 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 07 19:02:12.055445 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:02:12.055385 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-node-exporter-tls podName:ff8991a0-e43b-429f-bba8-6afbd5dd08ee nodeName:}" failed. No retries permitted until 2026-07-07 19:02:12.555370962 +0000 UTC m=+102.747911012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-node-exporter-tls") pod "node-exporter-rvlhp" (UID: "ff8991a0-e43b-429f-bba8-6afbd5dd08ee") : secret "node-exporter-tls" not found Jul 07 19:02:12.055659 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.055619 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0f243a4-a647-4eb4-bdbf-16638f205522-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-222dv\" (UID: \"c0f243a4-a647-4eb4-bdbf-16638f205522\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv" Jul 07 19:02:12.055771 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.055697 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/db8101fa-7197-464d-b9a8-385f5629f7ff-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-cdfrm\" (UID: \"db8101fa-7197-464d-b9a8-385f5629f7ff\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:12.058574 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.058548 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.058686 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.058614 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0f243a4-a647-4eb4-bdbf-16638f205522-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-222dv\" (UID: \"c0f243a4-a647-4eb4-bdbf-16638f205522\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv" Jul 07 19:02:12.059084 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.059062 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0f243a4-a647-4eb4-bdbf-16638f205522-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-222dv\" (UID: \"c0f243a4-a647-4eb4-bdbf-16638f205522\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv" Jul 07 19:02:12.060239 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.060217 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/db8101fa-7197-464d-b9a8-385f5629f7ff-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-cdfrm\" (UID: \"db8101fa-7197-464d-b9a8-385f5629f7ff\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:12.068759 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.068729 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nnp6\" (UniqueName: \"kubernetes.io/projected/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-kube-api-access-6nnp6\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.070221 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.070166 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhqhl\" (UniqueName: \"kubernetes.io/projected/c0f243a4-a647-4eb4-bdbf-16638f205522-kube-api-access-rhqhl\") pod \"openshift-state-metrics-54cf96dfcf-222dv\" (UID: \"c0f243a4-a647-4eb4-bdbf-16638f205522\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv" Jul 07 19:02:12.070221 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.070164 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t92zc\" (UniqueName: \"kubernetes.io/projected/db8101fa-7197-464d-b9a8-385f5629f7ff-kube-api-access-t92zc\") pod \"kube-state-metrics-7c9dbb8b7d-cdfrm\" (UID: \"db8101fa-7197-464d-b9a8-385f5629f7ff\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:12.075680 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.075461 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv" Jul 07 19:02:12.308326 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.308236 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv"] Jul 07 19:02:12.310755 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:02:12.310727 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f243a4_a647_4eb4_bdbf_16638f205522.slice/crio-d6b090d28396bf9d7e08b5a0e88d2b92903e6647a3022d14d132c15aa4c0f5c8 WatchSource:0}: Error finding container d6b090d28396bf9d7e08b5a0e88d2b92903e6647a3022d14d132c15aa4c0f5c8: Status 404 returned error can't find the container with id d6b090d28396bf9d7e08b5a0e88d2b92903e6647a3022d14d132c15aa4c0f5c8 Jul 07 19:02:12.560408 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.560313 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-node-exporter-tls\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.560408 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.560385 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db8101fa-7197-464d-b9a8-385f5629f7ff-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-cdfrm\" (UID: \"db8101fa-7197-464d-b9a8-385f5629f7ff\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:12.562818 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.562789 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ff8991a0-e43b-429f-bba8-6afbd5dd08ee-node-exporter-tls\") pod \"node-exporter-rvlhp\" (UID: \"ff8991a0-e43b-429f-bba8-6afbd5dd08ee\") " pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.562897 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.562847 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/db8101fa-7197-464d-b9a8-385f5629f7ff-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-cdfrm\" (UID: \"db8101fa-7197-464d-b9a8-385f5629f7ff\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:12.690107 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.690073 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8m6sh" Jul 07 19:02:12.705386 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.705355 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" Jul 07 19:02:12.731855 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.731822 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rvlhp" Jul 07 19:02:12.745923 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:02:12.745889 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8991a0_e43b_429f_bba8_6afbd5dd08ee.slice/crio-9e4450e7583b86dd8725956a7f8a0cb4a2f0051520c542d8297fa4fad1233c57 WatchSource:0}: Error finding container 9e4450e7583b86dd8725956a7f8a0cb4a2f0051520c542d8297fa4fad1233c57: Status 404 returned error can't find the container with id 9e4450e7583b86dd8725956a7f8a0cb4a2f0051520c542d8297fa4fad1233c57 Jul 07 19:02:12.817965 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.817873 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hrh48" event={"ID":"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74","Type":"ContainerStarted","Data":"3ba69fb88a4074980c18bf9eb1aa9850061fbf55e9dd4e134b1dfe8889961331"} Jul 07 19:02:12.817965 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.817932 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hrh48" event={"ID":"34b59a25-fd9d-4ed1-befe-7a9d4ad5db74","Type":"ContainerStarted","Data":"74ea1d83b64296bcd10e595b794c6ba5ef3fbc6cd34c2e2f21a7046792c0f189"} Jul 07 19:02:12.818502 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.818208 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hrh48" Jul 07 19:02:12.820101 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.820075 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv" event={"ID":"c0f243a4-a647-4eb4-bdbf-16638f205522","Type":"ContainerStarted","Data":"885e6fa125f6498f6bf766039d204ee07d013359852cc9fe31e01fe0eeffd539"} Jul 07 19:02:12.820221 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.820129 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv" event={"ID":"c0f243a4-a647-4eb4-bdbf-16638f205522","Type":"ContainerStarted","Data":"7a8a03fe799ed8085e78a27ce0c4f5bc814ebff23c90744ed86cecac7820cbcd"} Jul 07 19:02:12.820221 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.820158 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv" event={"ID":"c0f243a4-a647-4eb4-bdbf-16638f205522","Type":"ContainerStarted","Data":"d6b090d28396bf9d7e08b5a0e88d2b92903e6647a3022d14d132c15aa4c0f5c8"} Jul 07 19:02:12.821382 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.821355 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvlhp" event={"ID":"ff8991a0-e43b-429f-bba8-6afbd5dd08ee","Type":"ContainerStarted","Data":"9e4450e7583b86dd8725956a7f8a0cb4a2f0051520c542d8297fa4fad1233c57"} Jul 07 19:02:12.839798 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.839744 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm"] Jul 07 19:02:12.842502 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:12.842440 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hrh48" podStartSLOduration=65.154475568 podStartE2EDuration="1m6.842420859s" totalImagesPullingTime="1.687945291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:12.841357168 +0000 UTC m=+103.033897235" watchObservedRunningTime="2026-07-07 19:02:12.842420859 +0000 UTC m=+103.034960931" Jul 07 19:02:12.842614 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:02:12.842535 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8101fa_7197_464d_b9a8_385f5629f7ff.slice/crio-7de93faf3820cb670bcccd77fc12f2a82978960b97aa2d6e1db5121741cd42fd WatchSource:0}: Error finding container 7de93faf3820cb670bcccd77fc12f2a82978960b97aa2d6e1db5121741cd42fd: Status 404 returned error can't find the container with id 7de93faf3820cb670bcccd77fc12f2a82978960b97aa2d6e1db5121741cd42fd Jul 07 19:02:13.826426 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:13.826380 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" event={"ID":"db8101fa-7197-464d-b9a8-385f5629f7ff","Type":"ContainerStarted","Data":"7de93faf3820cb670bcccd77fc12f2a82978960b97aa2d6e1db5121741cd42fd"} Jul 07 19:02:14.857083 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.857032 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr"] Jul 07 19:02:14.890493 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.890459 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr"] Jul 07 19:02:14.890619 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.890590 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:14.892868 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.892847 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 07 19:02:14.895565 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.895541 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 07 19:02:14.895729 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.895707 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 07 19:02:14.895944 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.895710 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 07 19:02:14.895944 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.895789 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gn6ks\"" Jul 07 19:02:14.896127 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.895789 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 07 19:02:14.896127 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.895841 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-d6ar0vocfnjkq\"" Jul 07 19:02:14.983367 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.983342 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/213c5b68-851c-48c2-922b-120d4c007aed-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:14.983488 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.983376 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/213c5b68-851c-48c2-922b-120d4c007aed-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:14.983488 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.983410 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/213c5b68-851c-48c2-922b-120d4c007aed-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:14.983488 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.983463 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/213c5b68-851c-48c2-922b-120d4c007aed-metrics-client-ca\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:14.983649 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.983528 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/213c5b68-851c-48c2-922b-120d4c007aed-secret-thanos-querier-tls\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:14.983649 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.983575 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/213c5b68-851c-48c2-922b-120d4c007aed-secret-grpc-tls\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:14.983649 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.983595 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfgcz\" (UniqueName: \"kubernetes.io/projected/213c5b68-851c-48c2-922b-120d4c007aed-kube-api-access-gfgcz\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:14.983649 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.983616 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/213c5b68-851c-48c2-922b-120d4c007aed-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:14.992721 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.992674 2568 patch_prober.go:28] interesting pod/image-registry-76f5b666df-rl2xc container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 07 19:02:14.992846 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:14.992751 2568 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" podUID="9a2baf66-4d94-4bca-b8c8-7d64df15ad0e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 19:02:15.084782 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.084749 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/213c5b68-851c-48c2-922b-120d4c007aed-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:15.084949 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.084788 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/213c5b68-851c-48c2-922b-120d4c007aed-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:15.084949 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.084834 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/213c5b68-851c-48c2-922b-120d4c007aed-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:15.084949 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.084862 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/213c5b68-851c-48c2-922b-120d4c007aed-metrics-client-ca\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:15.084949 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.084914 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/213c5b68-851c-48c2-922b-120d4c007aed-secret-thanos-querier-tls\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:15.085185 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.084963 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/213c5b68-851c-48c2-922b-120d4c007aed-secret-grpc-tls\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:15.085185 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.084990 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfgcz\" (UniqueName: \"kubernetes.io/projected/213c5b68-851c-48c2-922b-120d4c007aed-kube-api-access-gfgcz\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:15.085185 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.085024 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/213c5b68-851c-48c2-922b-120d4c007aed-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:15.085718 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.085695 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/213c5b68-851c-48c2-922b-120d4c007aed-metrics-client-ca\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:15.087802 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.087770 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/213c5b68-851c-48c2-922b-120d4c007aed-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:15.087898 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.087880 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/213c5b68-851c-48c2-922b-120d4c007aed-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:15.087967 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.087946 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/213c5b68-851c-48c2-922b-120d4c007aed-secret-thanos-querier-tls\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:15.089291 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.089225 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/213c5b68-851c-48c2-922b-120d4c007aed-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:15.089437 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.089382 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/213c5b68-851c-48c2-922b-120d4c007aed-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:15.089870 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.089848 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/213c5b68-851c-48c2-922b-120d4c007aed-secret-grpc-tls\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:15.098096 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.098039 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfgcz\" (UniqueName: \"kubernetes.io/projected/213c5b68-851c-48c2-922b-120d4c007aed-kube-api-access-gfgcz\") pod \"thanos-querier-69b8b8d8d4-qfsbr\" (UID: \"213c5b68-851c-48c2-922b-120d4c007aed\") " pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:15.200832 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.200771 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:15.363694 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.363663 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr"] Jul 07 19:02:15.366034 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:02:15.366003 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213c5b68_851c_48c2_922b_120d4c007aed.slice/crio-db64d86f8f793e62b62b18cd34b856dadfe67bbfa5c83b858c7c867863127f7a WatchSource:0}: Error finding container db64d86f8f793e62b62b18cd34b856dadfe67bbfa5c83b858c7c867863127f7a: Status 404 returned error can't find the container with id db64d86f8f793e62b62b18cd34b856dadfe67bbfa5c83b858c7c867863127f7a Jul 07 19:02:15.837829 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.837792 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv" event={"ID":"c0f243a4-a647-4eb4-bdbf-16638f205522","Type":"ContainerStarted","Data":"24e37e9297ba350e3ead4cff6a35de8684448c177e08757cc8734b4e7217f665"} Jul 07 19:02:15.838936 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.838906 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" event={"ID":"213c5b68-851c-48c2-922b-120d4c007aed","Type":"ContainerStarted","Data":"db64d86f8f793e62b62b18cd34b856dadfe67bbfa5c83b858c7c867863127f7a"} Jul 07 19:02:15.840281 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.840259 2568 generic.go:356] "Generic (PLEG): container finished" podID="ff8991a0-e43b-429f-bba8-6afbd5dd08ee" containerID="3c2d733f36583adbcf7504a961900ebee9cc3dbd5a601b936a15a868e51a9ba4" exitCode=0 Jul 07 19:02:15.840364 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.840338 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvlhp" event={"ID":"ff8991a0-e43b-429f-bba8-6afbd5dd08ee","Type":"ContainerDied","Data":"3c2d733f36583adbcf7504a961900ebee9cc3dbd5a601b936a15a868e51a9ba4"} Jul 07 19:02:15.842399 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.842369 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" event={"ID":"db8101fa-7197-464d-b9a8-385f5629f7ff","Type":"ContainerStarted","Data":"dd872fefbd9947579818151ec4fd98b8f1be5b6ce316364fe807c875e8eea17b"} Jul 07 19:02:15.842498 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.842428 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" event={"ID":"db8101fa-7197-464d-b9a8-385f5629f7ff","Type":"ContainerStarted","Data":"a95d910b389c866756c7467725373ee066914e649167f389bf572d384e622db3"} Jul 07 19:02:15.842498 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.842459 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" event={"ID":"db8101fa-7197-464d-b9a8-385f5629f7ff","Type":"ContainerStarted","Data":"47c87eac8dc54bfa04bdccef188a985e1f1a2d65317a9f19145f02be49584ac4"} Jul 07 19:02:15.862458 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.862416 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-222dv" podStartSLOduration=2.417155026 podStartE2EDuration="4.862398632s" totalImagesPullingTime="2.445243606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:15.860726898 +0000 UTC m=+106.053266965" watchObservedRunningTime="2026-07-07 19:02:15.862398632 +0000 UTC m=+106.054938702" Jul 07 19:02:15.923700 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:15.923656 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-cdfrm" podStartSLOduration=2.858926441 podStartE2EDuration="4.923635669s" totalImagesPullingTime="2.064709228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:15.885087031 +0000 UTC m=+106.077627097" watchObservedRunningTime="2026-07-07 19:02:15.923635669 +0000 UTC m=+106.116175736" Jul 07 19:02:16.504062 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:16.504008 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-k98wx"] Jul 07 19:02:16.518077 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:16.517898 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-k98wx"] Jul 07 19:02:16.518077 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:16.518071 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98wx" Jul 07 19:02:16.520884 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:16.520444 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 07 19:02:16.520884 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:16.520731 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-smrh8\"" Jul 07 19:02:16.598912 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:16.598859 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khjsf\" (UniqueName: \"kubernetes.io/projected/890a28fe-4573-40f6-8336-eabf828bab94-kube-api-access-khjsf\") pod \"monitoring-plugin-d5dcff7cf-k98wx\" (UID: \"890a28fe-4573-40f6-8336-eabf828bab94\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98wx" Jul 07 19:02:16.598912 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:16.598913 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/890a28fe-4573-40f6-8336-eabf828bab94-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-k98wx\" (UID: \"890a28fe-4573-40f6-8336-eabf828bab94\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98wx" Jul 07 19:02:16.700664 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:16.700577 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khjsf\" (UniqueName: \"kubernetes.io/projected/890a28fe-4573-40f6-8336-eabf828bab94-kube-api-access-khjsf\") pod \"monitoring-plugin-d5dcff7cf-k98wx\" (UID: \"890a28fe-4573-40f6-8336-eabf828bab94\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98wx" Jul 07 19:02:16.700846 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:16.700680 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/890a28fe-4573-40f6-8336-eabf828bab94-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-k98wx\" (UID: \"890a28fe-4573-40f6-8336-eabf828bab94\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98wx" Jul 07 19:02:16.700846 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:02:16.700816 2568 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 07 19:02:16.700946 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:02:16.700891 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/890a28fe-4573-40f6-8336-eabf828bab94-monitoring-plugin-cert podName:890a28fe-4573-40f6-8336-eabf828bab94 nodeName:}" failed. No retries permitted until 2026-07-07 19:02:17.200869969 +0000 UTC m=+107.393410040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/890a28fe-4573-40f6-8336-eabf828bab94-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-k98wx" (UID: "890a28fe-4573-40f6-8336-eabf828bab94") : secret "monitoring-plugin-cert" not found Jul 07 19:02:16.711547 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:16.711515 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khjsf\" (UniqueName: \"kubernetes.io/projected/890a28fe-4573-40f6-8336-eabf828bab94-kube-api-access-khjsf\") pod \"monitoring-plugin-d5dcff7cf-k98wx\" (UID: \"890a28fe-4573-40f6-8336-eabf828bab94\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98wx" Jul 07 19:02:16.851963 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:16.851865 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvlhp" event={"ID":"ff8991a0-e43b-429f-bba8-6afbd5dd08ee","Type":"ContainerStarted","Data":"d9fd16a211d484c0e2c902efe19bfaad8922e0ec1aaebd6521243ed6a672e946"} Jul 07 19:02:16.852183 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:16.851977 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rvlhp" event={"ID":"ff8991a0-e43b-429f-bba8-6afbd5dd08ee","Type":"ContainerStarted","Data":"5128c60d5a66125ffc31f18f6d1a217bfea0d0447870be0579e6890c3ae88c57"} Jul 07 19:02:16.875168 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:16.875115 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rvlhp" podStartSLOduration=3.591920953 podStartE2EDuration="5.875099403s" totalImagesPullingTime="2.16322805s" totalInitContainerRuntime="119.9504ms" isStatefulPod=true podCreationTimestamp="2026-07-07 19:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:16.874017692 +0000 UTC m=+107.066557760" watchObservedRunningTime="2026-07-07 19:02:16.875099403 +0000 UTC m=+107.067639492" Jul 07 19:02:17.205537 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:17.205495 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/890a28fe-4573-40f6-8336-eabf828bab94-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-k98wx\" (UID: \"890a28fe-4573-40f6-8336-eabf828bab94\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98wx" Jul 07 19:02:17.208022 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:17.207995 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/890a28fe-4573-40f6-8336-eabf828bab94-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-k98wx\" (UID: \"890a28fe-4573-40f6-8336-eabf828bab94\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98wx" Jul 07 19:02:17.431171 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:17.431132 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98wx" Jul 07 19:02:17.750277 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:17.750247 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76f5b666df-rl2xc" Jul 07 19:02:17.991572 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:17.991539 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-k98wx"] Jul 07 19:02:17.995604 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:02:17.995444 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890a28fe_4573_40f6_8336_eabf828bab94.slice/crio-008b81444bc8d86fe70fd182472ec609bd5173b16f68ed69cf62d4e92bedb5a6 WatchSource:0}: Error finding container 008b81444bc8d86fe70fd182472ec609bd5173b16f68ed69cf62d4e92bedb5a6: Status 404 returned error can't find the container with id 008b81444bc8d86fe70fd182472ec609bd5173b16f68ed69cf62d4e92bedb5a6 Jul 07 19:02:18.862906 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:18.862865 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98wx" event={"ID":"890a28fe-4573-40f6-8336-eabf828bab94","Type":"ContainerStarted","Data":"008b81444bc8d86fe70fd182472ec609bd5173b16f68ed69cf62d4e92bedb5a6"} Jul 07 19:02:18.865322 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:18.865293 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" event={"ID":"213c5b68-851c-48c2-922b-120d4c007aed","Type":"ContainerStarted","Data":"d0e9e2b8b4a5a7d389bb80fed712428680d7c69dbc75686155419289421c0bbc"} Jul 07 19:02:18.865450 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:18.865353 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" event={"ID":"213c5b68-851c-48c2-922b-120d4c007aed","Type":"ContainerStarted","Data":"26fe6da33ab366306fcf5513ff95d90aac70cb14c1d92adbd45ae62b252b9ea9"} Jul 07 19:02:18.865450 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:18.865386 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" event={"ID":"213c5b68-851c-48c2-922b-120d4c007aed","Type":"ContainerStarted","Data":"438f22596795e61c25b9bbf4699feddf6987a1cf190904955773359654c62fc9"} Jul 07 19:02:19.871074 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:19.870946 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98wx" event={"ID":"890a28fe-4573-40f6-8336-eabf828bab94","Type":"ContainerStarted","Data":"f181400025be60f0d74a77212cf482c3353c67dd4c5b04e1e59f968a86b8cf90"} Jul 07 19:02:19.871488 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:19.871080 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98wx" Jul 07 19:02:19.874353 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:19.874321 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" event={"ID":"213c5b68-851c-48c2-922b-120d4c007aed","Type":"ContainerStarted","Data":"1be94e53333778319334741d0bab8cc982622dd62aaee7c8f4f6fd905267564e"} Jul 07 19:02:19.874478 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:19.874394 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" event={"ID":"213c5b68-851c-48c2-922b-120d4c007aed","Type":"ContainerStarted","Data":"0d2c5d5b8557f1831743ab9dc9a043a9ba8c51c032a240dbc91f4001e3ad3d77"} Jul 07 19:02:19.874478 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:19.874435 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" event={"ID":"213c5b68-851c-48c2-922b-120d4c007aed","Type":"ContainerStarted","Data":"1c53532914b815b7cb9067e6509adea4a68d75ac345d6954f0f2497d5d3561d9"} Jul 07 19:02:19.874561 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:19.874478 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:19.876626 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:19.876603 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98wx" Jul 07 19:02:19.897888 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:19.897845 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-k98wx" podStartSLOduration=2.300333716 podStartE2EDuration="3.89783189s" totalImagesPullingTime="1.597498174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:02:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:19.896728204 +0000 UTC m=+110.089268271" watchObservedRunningTime="2026-07-07 19:02:19.89783189 +0000 UTC m=+110.090371957" Jul 07 19:02:19.931457 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:19.931407 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" podStartSLOduration=2.433872015 podStartE2EDuration="5.931392487s" totalImagesPullingTime="3.497520472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:02:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:19.930839671 +0000 UTC m=+110.123379741" watchObservedRunningTime="2026-07-07 19:02:19.931392487 +0000 UTC m=+110.123932553" Jul 07 19:02:20.155275 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:20.155243 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-9cbxm"] Jul 07 19:02:20.157586 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:20.157570 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-9cbxm" Jul 07 19:02:20.160106 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:20.160086 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-4v776\"" Jul 07 19:02:20.160336 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:20.160321 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 19:02:20.170779 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:20.170755 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 19:02:20.185455 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:20.185432 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-9cbxm"] Jul 07 19:02:20.237254 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:20.237220 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2q59\" (UniqueName: \"kubernetes.io/projected/6ab43ed2-15f4-4bbd-ab1d-a06533c1afd7-kube-api-access-f2q59\") pod \"downloads-685bfb5f64-9cbxm\" (UID: \"6ab43ed2-15f4-4bbd-ab1d-a06533c1afd7\") " pod="openshift-console/downloads-685bfb5f64-9cbxm" Jul 07 19:02:20.338319 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:20.338287 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2q59\" (UniqueName: \"kubernetes.io/projected/6ab43ed2-15f4-4bbd-ab1d-a06533c1afd7-kube-api-access-f2q59\") pod \"downloads-685bfb5f64-9cbxm\" (UID: \"6ab43ed2-15f4-4bbd-ab1d-a06533c1afd7\") " pod="openshift-console/downloads-685bfb5f64-9cbxm" Jul 07 19:02:20.356400 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:20.356365 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2q59\" (UniqueName: \"kubernetes.io/projected/6ab43ed2-15f4-4bbd-ab1d-a06533c1afd7-kube-api-access-f2q59\") pod \"downloads-685bfb5f64-9cbxm\" (UID: \"6ab43ed2-15f4-4bbd-ab1d-a06533c1afd7\") " pod="openshift-console/downloads-685bfb5f64-9cbxm" Jul 07 19:02:20.466688 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:20.466601 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-9cbxm" Jul 07 19:02:20.609302 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:20.609279 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-9cbxm"] Jul 07 19:02:20.611083 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:02:20.611039 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab43ed2_15f4_4bbd_ab1d_a06533c1afd7.slice/crio-7b9ceb4ec65a155f42a4741ccdc9f173bbaaccf81d12c1380016d037aac52e72 WatchSource:0}: Error finding container 7b9ceb4ec65a155f42a4741ccdc9f173bbaaccf81d12c1380016d037aac52e72: Status 404 returned error can't find the container with id 7b9ceb4ec65a155f42a4741ccdc9f173bbaaccf81d12c1380016d037aac52e72 Jul 07 19:02:20.879419 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:20.879386 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-9cbxm" event={"ID":"6ab43ed2-15f4-4bbd-ab1d-a06533c1afd7","Type":"ContainerStarted","Data":"7b9ceb4ec65a155f42a4741ccdc9f173bbaaccf81d12c1380016d037aac52e72"} Jul 07 19:02:22.829414 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:22.829383 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hrh48" Jul 07 19:02:25.885781 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:25.885751 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-69b8b8d8d4-qfsbr" Jul 07 19:02:33.801063 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.801018 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-597dbc445b-d2lwr"] Jul 07 19:02:33.807261 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.807227 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.811795 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.811767 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6wjfm\"" Jul 07 19:02:33.811795 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.811788 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 19:02:33.812004 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.811830 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 19:02:33.812259 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.812243 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 19:02:33.812350 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.812315 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 19:02:33.812419 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.812246 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 19:02:33.819962 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.819940 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-597dbc445b-d2lwr"] Jul 07 19:02:33.822731 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.822705 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 19:02:33.866710 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.866672 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df99a209-c465-4902-949e-7fa54e0788cf-console-serving-cert\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.866899 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.866759 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-trusted-ca-bundle\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.866899 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.866808 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz5rm\" (UniqueName: \"kubernetes.io/projected/df99a209-c465-4902-949e-7fa54e0788cf-kube-api-access-wz5rm\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.866899 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.866852 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df99a209-c465-4902-949e-7fa54e0788cf-console-oauth-config\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.866899 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.866892 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-console-config\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.867117 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.866916 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-service-ca\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.867117 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.866958 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-oauth-serving-cert\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.967494 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.967456 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df99a209-c465-4902-949e-7fa54e0788cf-console-serving-cert\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.967676 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.967551 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-trusted-ca-bundle\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.967676 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.967606 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz5rm\" (UniqueName: \"kubernetes.io/projected/df99a209-c465-4902-949e-7fa54e0788cf-kube-api-access-wz5rm\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.967676 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.967649 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df99a209-c465-4902-949e-7fa54e0788cf-console-oauth-config\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.967833 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.967688 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-console-config\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.967833 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.967713 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-service-ca\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.967833 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.967741 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-oauth-serving-cert\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.968605 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.968574 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-oauth-serving-cert\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.968731 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.968669 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-trusted-ca-bundle\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.968731 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.968669 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-service-ca\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.968848 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.968740 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-console-config\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.970488 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.970467 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df99a209-c465-4902-949e-7fa54e0788cf-console-oauth-config\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.970650 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.970632 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df99a209-c465-4902-949e-7fa54e0788cf-console-serving-cert\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:33.979032 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:33.979006 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz5rm\" (UniqueName: \"kubernetes.io/projected/df99a209-c465-4902-949e-7fa54e0788cf-kube-api-access-wz5rm\") pod \"console-597dbc445b-d2lwr\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:34.119295 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:34.119209 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:38.424317 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:38.424290 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-597dbc445b-d2lwr"] Jul 07 19:02:38.436213 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:02:38.436187 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf99a209_c465_4902_949e_7fa54e0788cf.slice/crio-35c246fbc4d62dd2065377fe8618e98622704d84f3d3b10cd6bbfa2c9b4df740 WatchSource:0}: Error finding container 35c246fbc4d62dd2065377fe8618e98622704d84f3d3b10cd6bbfa2c9b4df740: Status 404 returned error can't find the container with id 35c246fbc4d62dd2065377fe8618e98622704d84f3d3b10cd6bbfa2c9b4df740 Jul 07 19:02:38.960075 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:38.959810 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-9cbxm" event={"ID":"6ab43ed2-15f4-4bbd-ab1d-a06533c1afd7","Type":"ContainerStarted","Data":"a8550256869653a86b86c07d00645f2269edd88273b40d34e55f9be2dcf4505e"} Jul 07 19:02:38.960271 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:38.960140 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-9cbxm" Jul 07 19:02:38.962580 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:38.962544 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597dbc445b-d2lwr" event={"ID":"df99a209-c465-4902-949e-7fa54e0788cf","Type":"ContainerStarted","Data":"35c246fbc4d62dd2065377fe8618e98622704d84f3d3b10cd6bbfa2c9b4df740"} Jul 07 19:02:38.970089 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:38.970039 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-9cbxm" Jul 07 19:02:38.981318 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:38.980613 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-9cbxm" podStartSLOduration=1.193135271 podStartE2EDuration="18.980595146s" totalImagesPullingTime="17.787459875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:38.979372266 +0000 UTC m=+129.171912333" watchObservedRunningTime="2026-07-07 19:02:38.980595146 +0000 UTC m=+129.173135214" Jul 07 19:02:39.115448 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:39.115408 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs\") pod \"network-metrics-daemon-rfz5t\" (UID: \"848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c\") " pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:02:39.118360 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:39.118331 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c-metrics-certs\") pod \"network-metrics-daemon-rfz5t\" (UID: \"848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c\") " pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:02:39.137701 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:39.137670 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-84rbs\"" Jul 07 19:02:39.145786 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:39.145743 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rfz5t" Jul 07 19:02:39.331219 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:39.331138 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rfz5t"] Jul 07 19:02:39.334307 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:02:39.334262 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod848ad3ea_2ce4_44f2_b1ad_a5eae7f3a40c.slice/crio-6568e15da1b678e96d39d8f591061ef007351a88fa8e4984a818472f2506241f WatchSource:0}: Error finding container 6568e15da1b678e96d39d8f591061ef007351a88fa8e4984a818472f2506241f: Status 404 returned error can't find the container with id 6568e15da1b678e96d39d8f591061ef007351a88fa8e4984a818472f2506241f Jul 07 19:02:39.970024 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:39.969968 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rfz5t" event={"ID":"848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c","Type":"ContainerStarted","Data":"6568e15da1b678e96d39d8f591061ef007351a88fa8e4984a818472f2506241f"} Jul 07 19:02:40.977753 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:40.977711 2568 generic.go:356] "Generic (PLEG): container finished" podID="7fcae971-26d4-43f1-aa35-179c080f18bc" containerID="9fab801178950cc4928dea2a002faa78b30f6e4df3deb9d176b38776514692d7" exitCode=0 Jul 07 19:02:40.978251 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:40.977784 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mz9pd" event={"ID":"7fcae971-26d4-43f1-aa35-179c080f18bc","Type":"ContainerDied","Data":"9fab801178950cc4928dea2a002faa78b30f6e4df3deb9d176b38776514692d7"} Jul 07 19:02:40.978342 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:40.978322 2568 scope.go:117] "RemoveContainer" containerID="9fab801178950cc4928dea2a002faa78b30f6e4df3deb9d176b38776514692d7" Jul 07 19:02:42.989511 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:42.989463 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-mz9pd" event={"ID":"7fcae971-26d4-43f1-aa35-179c080f18bc","Type":"ContainerStarted","Data":"8095552a038a4da72c6edf3763312839fcfd3e1ca4eef86ded912d8c5ffdc269"} Jul 07 19:02:42.992193 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:42.992157 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rfz5t" event={"ID":"848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c","Type":"ContainerStarted","Data":"a89c2a7232a3a5b85593b75e0ed4c2d99daae0cf2af34f91f928a123fb6dd0d9"} Jul 07 19:02:42.992322 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:42.992217 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rfz5t" event={"ID":"848ad3ea-2ce4-44f2-b1ad-a5eae7f3a40c","Type":"ContainerStarted","Data":"629ec0478c5f392681336a054cecf43f34f2d8e0ad729722590a175372479b6a"} Jul 07 19:02:42.995675 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:42.995603 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597dbc445b-d2lwr" event={"ID":"df99a209-c465-4902-949e-7fa54e0788cf","Type":"ContainerStarted","Data":"21bacf2f1d6dd80658f7cfd6b5c0d9114353837f8a1eb58797f15d21821152e1"} Jul 07 19:02:43.061741 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:43.061166 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-597dbc445b-d2lwr" podStartSLOduration=5.960036885 podStartE2EDuration="10.061143209s" totalImagesPullingTime="4.101106324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:02:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:43.040559552 +0000 UTC m=+133.233099619" watchObservedRunningTime="2026-07-07 19:02:43.061143209 +0000 UTC m=+133.253683278" Jul 07 19:02:43.062126 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:43.062080 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rfz5t" podStartSLOduration=128.865463757 podStartE2EDuration="2m12.062064049s" totalImagesPullingTime="3.196600292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:43.061029612 +0000 UTC m=+133.253569675" watchObservedRunningTime="2026-07-07 19:02:43.062064049 +0000 UTC m=+133.254604119" Jul 07 19:02:44.120105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:44.120063 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:44.120578 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:44.120146 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:44.125839 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:44.125806 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:02:45.009094 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:02:45.009060 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:03:00.071600 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:03:00.071568 2568 generic.go:356] "Generic (PLEG): container finished" podID="32659884-db20-426c-be99-476c7718c18d" containerID="19df1d61e18a4c2d6688280654ae92e88771e3a4a489f7420199f407d764b6ec" exitCode=0 Jul 07 19:03:00.072020 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:03:00.071645 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" event={"ID":"32659884-db20-426c-be99-476c7718c18d","Type":"ContainerDied","Data":"19df1d61e18a4c2d6688280654ae92e88771e3a4a489f7420199f407d764b6ec"} Jul 07 19:03:00.072020 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:03:00.071976 2568 scope.go:117] "RemoveContainer" containerID="19df1d61e18a4c2d6688280654ae92e88771e3a4a489f7420199f407d764b6ec" Jul 07 19:03:01.077323 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:03:01.077285 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-j7h8x" event={"ID":"32659884-db20-426c-be99-476c7718c18d","Type":"ContainerStarted","Data":"d67b897747e323c753feba14be65aabbc0c0dc9d62b63b1e3a7ee8bfa711eb0d"} Jul 07 19:03:52.224093 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:03:52.223997 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-597dbc445b-d2lwr"] Jul 07 19:04:17.244631 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.244581 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-597dbc445b-d2lwr" podUID="df99a209-c465-4902-949e-7fa54e0788cf" containerName="console" containerID="cri-o://21bacf2f1d6dd80658f7cfd6b5c0d9114353837f8a1eb58797f15d21821152e1" gracePeriod=15 Jul 07 19:04:17.365640 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.365608 2568 generic.go:356] "Generic (PLEG): container finished" podID="df99a209-c465-4902-949e-7fa54e0788cf" containerID="21bacf2f1d6dd80658f7cfd6b5c0d9114353837f8a1eb58797f15d21821152e1" exitCode=0 Jul 07 19:04:17.365780 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.365681 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597dbc445b-d2lwr" event={"ID":"df99a209-c465-4902-949e-7fa54e0788cf","Type":"ContainerDied","Data":"21bacf2f1d6dd80658f7cfd6b5c0d9114353837f8a1eb58797f15d21821152e1"} Jul 07 19:04:17.491210 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.491184 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:04:17.655092 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.655029 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-console-config\") pod \"df99a209-c465-4902-949e-7fa54e0788cf\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " Jul 07 19:04:17.655279 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.655116 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-trusted-ca-bundle\") pod \"df99a209-c465-4902-949e-7fa54e0788cf\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " Jul 07 19:04:17.655279 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.655167 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wz5rm\" (UniqueName: \"kubernetes.io/projected/df99a209-c465-4902-949e-7fa54e0788cf-kube-api-access-wz5rm\") pod \"df99a209-c465-4902-949e-7fa54e0788cf\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " Jul 07 19:04:17.655279 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.655207 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df99a209-c465-4902-949e-7fa54e0788cf-console-oauth-config\") pod \"df99a209-c465-4902-949e-7fa54e0788cf\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " Jul 07 19:04:17.655279 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.655247 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-oauth-serving-cert\") pod \"df99a209-c465-4902-949e-7fa54e0788cf\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " Jul 07 19:04:17.655483 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.655279 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-service-ca\") pod \"df99a209-c465-4902-949e-7fa54e0788cf\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " Jul 07 19:04:17.655483 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.655314 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df99a209-c465-4902-949e-7fa54e0788cf-console-serving-cert\") pod \"df99a209-c465-4902-949e-7fa54e0788cf\" (UID: \"df99a209-c465-4902-949e-7fa54e0788cf\") " Jul 07 19:04:17.655584 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.655487 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-console-config" (OuterVolumeSpecName: "console-config") pod "df99a209-c465-4902-949e-7fa54e0788cf" (UID: "df99a209-c465-4902-949e-7fa54e0788cf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:04:17.655640 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.655600 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "df99a209-c465-4902-949e-7fa54e0788cf" (UID: "df99a209-c465-4902-949e-7fa54e0788cf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:04:17.655807 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.655765 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "df99a209-c465-4902-949e-7fa54e0788cf" (UID: "df99a209-c465-4902-949e-7fa54e0788cf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:04:17.655916 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.655837 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-service-ca" (OuterVolumeSpecName: "service-ca") pod "df99a209-c465-4902-949e-7fa54e0788cf" (UID: "df99a209-c465-4902-949e-7fa54e0788cf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:04:17.657599 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.657568 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df99a209-c465-4902-949e-7fa54e0788cf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "df99a209-c465-4902-949e-7fa54e0788cf" (UID: "df99a209-c465-4902-949e-7fa54e0788cf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:04:17.657680 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.657629 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df99a209-c465-4902-949e-7fa54e0788cf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "df99a209-c465-4902-949e-7fa54e0788cf" (UID: "df99a209-c465-4902-949e-7fa54e0788cf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:04:17.657680 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.657632 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df99a209-c465-4902-949e-7fa54e0788cf-kube-api-access-wz5rm" (OuterVolumeSpecName: "kube-api-access-wz5rm") pod "df99a209-c465-4902-949e-7fa54e0788cf" (UID: "df99a209-c465-4902-949e-7fa54e0788cf"). InnerVolumeSpecName "kube-api-access-wz5rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:04:17.756119 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.756080 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-trusted-ca-bundle\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:04:17.756119 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.756113 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz5rm\" (UniqueName: \"kubernetes.io/projected/df99a209-c465-4902-949e-7fa54e0788cf-kube-api-access-wz5rm\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:04:17.756119 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.756123 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/df99a209-c465-4902-949e-7fa54e0788cf-console-oauth-config\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:04:17.756344 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.756132 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-oauth-serving-cert\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:04:17.756344 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.756142 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-service-ca\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:04:17.756344 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.756150 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/df99a209-c465-4902-949e-7fa54e0788cf-console-serving-cert\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:04:17.756344 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:17.756158 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/df99a209-c465-4902-949e-7fa54e0788cf-console-config\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:04:18.371277 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:18.371244 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597dbc445b-d2lwr" event={"ID":"df99a209-c465-4902-949e-7fa54e0788cf","Type":"ContainerDied","Data":"35c246fbc4d62dd2065377fe8618e98622704d84f3d3b10cd6bbfa2c9b4df740"} Jul 07 19:04:18.371670 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:18.371301 2568 scope.go:117] "RemoveContainer" containerID="21bacf2f1d6dd80658f7cfd6b5c0d9114353837f8a1eb58797f15d21821152e1" Jul 07 19:04:18.371670 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:18.371256 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597dbc445b-d2lwr" Jul 07 19:04:18.392772 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:18.392744 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-597dbc445b-d2lwr"] Jul 07 19:04:18.397185 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:18.397160 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-597dbc445b-d2lwr"] Jul 07 19:04:20.328320 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:04:20.328286 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df99a209-c465-4902-949e-7fa54e0788cf" path="/var/lib/kubelet/pods/df99a209-c465-4902-949e-7fa54e0788cf/volumes" Jul 07 19:05:21.693220 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:21.693183 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l"] Jul 07 19:05:21.693705 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:21.693532 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df99a209-c465-4902-949e-7fa54e0788cf" containerName="console" Jul 07 19:05:21.693705 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:21.693550 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="df99a209-c465-4902-949e-7fa54e0788cf" containerName="console" Jul 07 19:05:21.693705 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:21.693639 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="df99a209-c465-4902-949e-7fa54e0788cf" containerName="console" Jul 07 19:05:21.695708 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:21.695692 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l" Jul 07 19:05:21.698019 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:21.697994 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 19:05:21.698019 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:21.698004 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 19:05:21.698502 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:21.698488 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kgw6h\"" Jul 07 19:05:21.706412 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:21.706394 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l"] Jul 07 19:05:21.775718 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:21.775680 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57qqf\" (UniqueName: \"kubernetes.io/projected/823b24e7-baf2-451a-aab7-c8020f016a6b-kube-api-access-57qqf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l\" (UID: \"823b24e7-baf2-451a-aab7-c8020f016a6b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l" Jul 07 19:05:21.775894 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:21.775724 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/823b24e7-baf2-451a-aab7-c8020f016a6b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l\" (UID: \"823b24e7-baf2-451a-aab7-c8020f016a6b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l" Jul 07 19:05:21.775894 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:21.775762 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/823b24e7-baf2-451a-aab7-c8020f016a6b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l\" (UID: \"823b24e7-baf2-451a-aab7-c8020f016a6b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l" Jul 07 19:05:21.876851 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:21.876813 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57qqf\" (UniqueName: \"kubernetes.io/projected/823b24e7-baf2-451a-aab7-c8020f016a6b-kube-api-access-57qqf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l\" (UID: \"823b24e7-baf2-451a-aab7-c8020f016a6b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l" Jul 07 19:05:21.877057 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:21.876866 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/823b24e7-baf2-451a-aab7-c8020f016a6b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l\" (UID: \"823b24e7-baf2-451a-aab7-c8020f016a6b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l" Jul 07 19:05:21.877057 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:21.876922 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/823b24e7-baf2-451a-aab7-c8020f016a6b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l\" (UID: \"823b24e7-baf2-451a-aab7-c8020f016a6b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l" Jul 07 19:05:21.877287 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:21.877267 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/823b24e7-baf2-451a-aab7-c8020f016a6b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l\" (UID: \"823b24e7-baf2-451a-aab7-c8020f016a6b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l" Jul 07 19:05:21.877322 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:21.877310 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/823b24e7-baf2-451a-aab7-c8020f016a6b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l\" (UID: \"823b24e7-baf2-451a-aab7-c8020f016a6b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l" Jul 07 19:05:21.887644 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:21.887622 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57qqf\" (UniqueName: \"kubernetes.io/projected/823b24e7-baf2-451a-aab7-c8020f016a6b-kube-api-access-57qqf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l\" (UID: \"823b24e7-baf2-451a-aab7-c8020f016a6b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l" Jul 07 19:05:22.005517 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:22.005435 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l" Jul 07 19:05:22.135984 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:22.135950 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l"] Jul 07 19:05:22.139213 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:05:22.139186 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823b24e7_baf2_451a_aab7_c8020f016a6b.slice/crio-b6f7134fed18d9c12b8a78dcb8250815850ce16b3e89e19e23535f4a471f8757 WatchSource:0}: Error finding container b6f7134fed18d9c12b8a78dcb8250815850ce16b3e89e19e23535f4a471f8757: Status 404 returned error can't find the container with id b6f7134fed18d9c12b8a78dcb8250815850ce16b3e89e19e23535f4a471f8757 Jul 07 19:05:22.610407 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:22.610370 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l" event={"ID":"823b24e7-baf2-451a-aab7-c8020f016a6b","Type":"ContainerStarted","Data":"b6f7134fed18d9c12b8a78dcb8250815850ce16b3e89e19e23535f4a471f8757"} Jul 07 19:05:31.402133 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:31.402098 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:05:31.403735 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:31.402243 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:05:31.408878 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:31.408859 2568 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 19:05:31.647937 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:31.647904 2568 generic.go:356] "Generic (PLEG): container finished" podID="823b24e7-baf2-451a-aab7-c8020f016a6b" containerID="5b8b9d7d9818893fdb009488c3e3d238f70a2208a0fc5b62b25fb1b0e4059157" exitCode=0 Jul 07 19:05:31.648118 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:31.647961 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l" event={"ID":"823b24e7-baf2-451a-aab7-c8020f016a6b","Type":"ContainerDied","Data":"5b8b9d7d9818893fdb009488c3e3d238f70a2208a0fc5b62b25fb1b0e4059157"} Jul 07 19:05:31.649091 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:31.649073 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:05:34.661849 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:34.661818 2568 generic.go:356] "Generic (PLEG): container finished" podID="823b24e7-baf2-451a-aab7-c8020f016a6b" containerID="463a739e414256cc514fa3613dc934d55ebf769eeb20f87b0ea82ed7865965aa" exitCode=0 Jul 07 19:05:34.662254 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:34.661875 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l" event={"ID":"823b24e7-baf2-451a-aab7-c8020f016a6b","Type":"ContainerDied","Data":"463a739e414256cc514fa3613dc934d55ebf769eeb20f87b0ea82ed7865965aa"} Jul 07 19:05:41.694062 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:41.694015 2568 generic.go:356] "Generic (PLEG): container finished" podID="823b24e7-baf2-451a-aab7-c8020f016a6b" containerID="4d269efde23654cd75db9e024946491afc0d84650f36702fbb0719988f58845c" exitCode=0 Jul 07 19:05:41.694431 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:41.694107 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l" event={"ID":"823b24e7-baf2-451a-aab7-c8020f016a6b","Type":"ContainerDied","Data":"4d269efde23654cd75db9e024946491afc0d84650f36702fbb0719988f58845c"} Jul 07 19:05:42.821801 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:42.821779 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l" Jul 07 19:05:42.948454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:42.948360 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/823b24e7-baf2-451a-aab7-c8020f016a6b-bundle\") pod \"823b24e7-baf2-451a-aab7-c8020f016a6b\" (UID: \"823b24e7-baf2-451a-aab7-c8020f016a6b\") " Jul 07 19:05:42.948454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:42.948427 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/823b24e7-baf2-451a-aab7-c8020f016a6b-util\") pod \"823b24e7-baf2-451a-aab7-c8020f016a6b\" (UID: \"823b24e7-baf2-451a-aab7-c8020f016a6b\") " Jul 07 19:05:42.948695 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:42.948462 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57qqf\" (UniqueName: \"kubernetes.io/projected/823b24e7-baf2-451a-aab7-c8020f016a6b-kube-api-access-57qqf\") pod \"823b24e7-baf2-451a-aab7-c8020f016a6b\" (UID: \"823b24e7-baf2-451a-aab7-c8020f016a6b\") " Jul 07 19:05:42.949187 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:42.949151 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823b24e7-baf2-451a-aab7-c8020f016a6b-bundle" (OuterVolumeSpecName: "bundle") pod "823b24e7-baf2-451a-aab7-c8020f016a6b" (UID: "823b24e7-baf2-451a-aab7-c8020f016a6b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:05:42.950709 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:42.950690 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823b24e7-baf2-451a-aab7-c8020f016a6b-kube-api-access-57qqf" (OuterVolumeSpecName: "kube-api-access-57qqf") pod "823b24e7-baf2-451a-aab7-c8020f016a6b" (UID: "823b24e7-baf2-451a-aab7-c8020f016a6b"). InnerVolumeSpecName "kube-api-access-57qqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:05:42.954643 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:42.954609 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/823b24e7-baf2-451a-aab7-c8020f016a6b-util" (OuterVolumeSpecName: "util") pod "823b24e7-baf2-451a-aab7-c8020f016a6b" (UID: "823b24e7-baf2-451a-aab7-c8020f016a6b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:05:43.049333 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:43.049288 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/823b24e7-baf2-451a-aab7-c8020f016a6b-util\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:05:43.049333 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:43.049326 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57qqf\" (UniqueName: \"kubernetes.io/projected/823b24e7-baf2-451a-aab7-c8020f016a6b-kube-api-access-57qqf\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:05:43.049333 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:43.049339 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/823b24e7-baf2-451a-aab7-c8020f016a6b-bundle\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:05:43.703858 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:43.703821 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l" event={"ID":"823b24e7-baf2-451a-aab7-c8020f016a6b","Type":"ContainerDied","Data":"b6f7134fed18d9c12b8a78dcb8250815850ce16b3e89e19e23535f4a471f8757"} Jul 07 19:05:43.703858 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:43.703854 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dkp2l" Jul 07 19:05:43.704082 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:43.703881 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6f7134fed18d9c12b8a78dcb8250815850ce16b3e89e19e23535f4a471f8757" Jul 07 19:05:50.174126 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.174084 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cdf8h"] Jul 07 19:05:50.174531 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.174390 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="823b24e7-baf2-451a-aab7-c8020f016a6b" containerName="pull" Jul 07 19:05:50.174531 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.174401 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="823b24e7-baf2-451a-aab7-c8020f016a6b" containerName="pull" Jul 07 19:05:50.174531 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.174414 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="823b24e7-baf2-451a-aab7-c8020f016a6b" containerName="util" Jul 07 19:05:50.174531 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.174419 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="823b24e7-baf2-451a-aab7-c8020f016a6b" containerName="util" Jul 07 19:05:50.174531 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.174431 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="823b24e7-baf2-451a-aab7-c8020f016a6b" containerName="extract" Jul 07 19:05:50.174531 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.174437 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="823b24e7-baf2-451a-aab7-c8020f016a6b" containerName="extract" Jul 07 19:05:50.174531 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.174489 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="823b24e7-baf2-451a-aab7-c8020f016a6b" containerName="extract" Jul 07 19:05:50.179769 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.179747 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cdf8h" Jul 07 19:05:50.183094 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.183072 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:05:50.183491 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.183472 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 07 19:05:50.183631 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.183612 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-lgtnx\"" Jul 07 19:05:50.201559 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.201528 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cdf8h"] Jul 07 19:05:50.208083 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.208039 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm55z\" (UniqueName: \"kubernetes.io/projected/3f1837eb-0097-46e5-907a-75e17ddb923a-kube-api-access-nm55z\") pod \"cert-manager-operator-controller-manager-55c444bf86-cdf8h\" (UID: \"3f1837eb-0097-46e5-907a-75e17ddb923a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cdf8h" Jul 07 19:05:50.208215 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.208130 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f1837eb-0097-46e5-907a-75e17ddb923a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cdf8h\" (UID: \"3f1837eb-0097-46e5-907a-75e17ddb923a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cdf8h" Jul 07 19:05:50.308534 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.308497 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f1837eb-0097-46e5-907a-75e17ddb923a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cdf8h\" (UID: \"3f1837eb-0097-46e5-907a-75e17ddb923a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cdf8h" Jul 07 19:05:50.308696 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.308588 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm55z\" (UniqueName: \"kubernetes.io/projected/3f1837eb-0097-46e5-907a-75e17ddb923a-kube-api-access-nm55z\") pod \"cert-manager-operator-controller-manager-55c444bf86-cdf8h\" (UID: \"3f1837eb-0097-46e5-907a-75e17ddb923a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cdf8h" Jul 07 19:05:50.308954 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.308916 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f1837eb-0097-46e5-907a-75e17ddb923a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cdf8h\" (UID: \"3f1837eb-0097-46e5-907a-75e17ddb923a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cdf8h" Jul 07 19:05:50.318997 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.318961 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm55z\" (UniqueName: \"kubernetes.io/projected/3f1837eb-0097-46e5-907a-75e17ddb923a-kube-api-access-nm55z\") pod \"cert-manager-operator-controller-manager-55c444bf86-cdf8h\" (UID: \"3f1837eb-0097-46e5-907a-75e17ddb923a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cdf8h" Jul 07 19:05:50.489515 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.489413 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cdf8h" Jul 07 19:05:50.626482 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.626450 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cdf8h"] Jul 07 19:05:50.629812 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:05:50.629781 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1837eb_0097_46e5_907a_75e17ddb923a.slice/crio-b45d0b79596fa8da87de53ec4793e014a9dd845d0ceac6f68d69cfa2477be4a5 WatchSource:0}: Error finding container b45d0b79596fa8da87de53ec4793e014a9dd845d0ceac6f68d69cfa2477be4a5: Status 404 returned error can't find the container with id b45d0b79596fa8da87de53ec4793e014a9dd845d0ceac6f68d69cfa2477be4a5 Jul 07 19:05:50.733640 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:50.733604 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cdf8h" event={"ID":"3f1837eb-0097-46e5-907a-75e17ddb923a","Type":"ContainerStarted","Data":"b45d0b79596fa8da87de53ec4793e014a9dd845d0ceac6f68d69cfa2477be4a5"} Jul 07 19:05:52.744571 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:52.744526 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cdf8h" event={"ID":"3f1837eb-0097-46e5-907a-75e17ddb923a","Type":"ContainerStarted","Data":"a0091af84045088c8f7806d59b613cc5fd3a92a1da742b2a8f54b5c88dc6acac"} Jul 07 19:05:52.769371 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:52.769315 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cdf8h" podStartSLOduration=1.056214475 podStartE2EDuration="2.76930047s" totalImagesPullingTime="1.713085995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:05:52.767415058 +0000 UTC m=+322.959955211" watchObservedRunningTime="2026-07-07 19:05:52.76930047 +0000 UTC m=+322.961840537" Jul 07 19:05:54.009691 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.009653 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb"] Jul 07 19:05:54.012264 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.012247 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb" Jul 07 19:05:54.016597 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.016563 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 19:05:54.016597 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.016585 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kgw6h\"" Jul 07 19:05:54.016771 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.016571 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 19:05:54.025595 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.025573 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb"] Jul 07 19:05:54.047445 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.047416 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae611325-f231-4121-83fb-4d2e01464658-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb\" (UID: \"ae611325-f231-4121-83fb-4d2e01464658\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb" Jul 07 19:05:54.047445 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.047445 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbr5s\" (UniqueName: \"kubernetes.io/projected/ae611325-f231-4121-83fb-4d2e01464658-kube-api-access-kbr5s\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb\" (UID: \"ae611325-f231-4121-83fb-4d2e01464658\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb" Jul 07 19:05:54.047628 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.047466 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae611325-f231-4121-83fb-4d2e01464658-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb\" (UID: \"ae611325-f231-4121-83fb-4d2e01464658\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb" Jul 07 19:05:54.148404 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.148373 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae611325-f231-4121-83fb-4d2e01464658-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb\" (UID: \"ae611325-f231-4121-83fb-4d2e01464658\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb" Jul 07 19:05:54.148404 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.148407 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbr5s\" (UniqueName: \"kubernetes.io/projected/ae611325-f231-4121-83fb-4d2e01464658-kube-api-access-kbr5s\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb\" (UID: \"ae611325-f231-4121-83fb-4d2e01464658\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb" Jul 07 19:05:54.148616 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.148431 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae611325-f231-4121-83fb-4d2e01464658-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb\" (UID: \"ae611325-f231-4121-83fb-4d2e01464658\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb" Jul 07 19:05:54.148812 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.148781 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae611325-f231-4121-83fb-4d2e01464658-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb\" (UID: \"ae611325-f231-4121-83fb-4d2e01464658\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb" Jul 07 19:05:54.148812 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.148804 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae611325-f231-4121-83fb-4d2e01464658-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb\" (UID: \"ae611325-f231-4121-83fb-4d2e01464658\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb" Jul 07 19:05:54.158824 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.158792 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbr5s\" (UniqueName: \"kubernetes.io/projected/ae611325-f231-4121-83fb-4d2e01464658-kube-api-access-kbr5s\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb\" (UID: \"ae611325-f231-4121-83fb-4d2e01464658\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb" Jul 07 19:05:54.321810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.321717 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb" Jul 07 19:05:54.458944 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.458915 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb"] Jul 07 19:05:54.461095 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:05:54.461059 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae611325_f231_4121_83fb_4d2e01464658.slice/crio-787ca7090ec164624f83c9c476bc45406bb006ab9fead5f680fe1531292ccb10 WatchSource:0}: Error finding container 787ca7090ec164624f83c9c476bc45406bb006ab9fead5f680fe1531292ccb10: Status 404 returned error can't find the container with id 787ca7090ec164624f83c9c476bc45406bb006ab9fead5f680fe1531292ccb10 Jul 07 19:05:54.754454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.754419 2568 generic.go:356] "Generic (PLEG): container finished" podID="ae611325-f231-4121-83fb-4d2e01464658" containerID="37e35a7b3b98cdd0e14c33867293803536e0e8e417e2dc05fe562bff649cea57" exitCode=0 Jul 07 19:05:54.754622 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.754507 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb" event={"ID":"ae611325-f231-4121-83fb-4d2e01464658","Type":"ContainerDied","Data":"37e35a7b3b98cdd0e14c33867293803536e0e8e417e2dc05fe562bff649cea57"} Jul 07 19:05:54.754622 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:54.754556 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb" event={"ID":"ae611325-f231-4121-83fb-4d2e01464658","Type":"ContainerStarted","Data":"787ca7090ec164624f83c9c476bc45406bb006ab9fead5f680fe1531292ccb10"} Jul 07 19:05:56.764771 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:56.764737 2568 generic.go:356] "Generic (PLEG): container finished" podID="ae611325-f231-4121-83fb-4d2e01464658" containerID="81dc276d15dbecbad770feb31a866001c13486c74fa0ae1333e9babe782ea462" exitCode=0 Jul 07 19:05:56.765194 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:56.764814 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb" event={"ID":"ae611325-f231-4121-83fb-4d2e01464658","Type":"ContainerDied","Data":"81dc276d15dbecbad770feb31a866001c13486c74fa0ae1333e9babe782ea462"} Jul 07 19:05:57.274319 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:57.274284 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bjxpv"] Jul 07 19:05:57.277599 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:57.277572 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bjxpv" Jul 07 19:05:57.280070 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:57.280034 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 19:05:57.280996 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:57.280977 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5msfl\"" Jul 07 19:05:57.281532 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:57.281512 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 19:05:57.289965 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:57.289944 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bjxpv"] Jul 07 19:05:57.378786 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:57.378751 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76w7p\" (UniqueName: \"kubernetes.io/projected/aca160df-fc11-4b89-b1be-31cb87ab03e7-kube-api-access-76w7p\") pod \"cert-manager-cainjector-559cdd8c87-bjxpv\" (UID: \"aca160df-fc11-4b89-b1be-31cb87ab03e7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bjxpv" Jul 07 19:05:57.378935 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:57.378810 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aca160df-fc11-4b89-b1be-31cb87ab03e7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bjxpv\" (UID: \"aca160df-fc11-4b89-b1be-31cb87ab03e7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bjxpv" Jul 07 19:05:57.479411 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:57.479375 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76w7p\" (UniqueName: \"kubernetes.io/projected/aca160df-fc11-4b89-b1be-31cb87ab03e7-kube-api-access-76w7p\") pod \"cert-manager-cainjector-559cdd8c87-bjxpv\" (UID: \"aca160df-fc11-4b89-b1be-31cb87ab03e7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bjxpv" Jul 07 19:05:57.479573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:57.479444 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aca160df-fc11-4b89-b1be-31cb87ab03e7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bjxpv\" (UID: \"aca160df-fc11-4b89-b1be-31cb87ab03e7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bjxpv" Jul 07 19:05:57.489595 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:57.489567 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/aca160df-fc11-4b89-b1be-31cb87ab03e7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bjxpv\" (UID: \"aca160df-fc11-4b89-b1be-31cb87ab03e7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bjxpv" Jul 07 19:05:57.490448 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:57.490420 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76w7p\" (UniqueName: \"kubernetes.io/projected/aca160df-fc11-4b89-b1be-31cb87ab03e7-kube-api-access-76w7p\") pod \"cert-manager-cainjector-559cdd8c87-bjxpv\" (UID: \"aca160df-fc11-4b89-b1be-31cb87ab03e7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bjxpv" Jul 07 19:05:57.587263 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:57.587173 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bjxpv" Jul 07 19:05:57.729515 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:57.729482 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bjxpv"] Jul 07 19:05:57.732611 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:05:57.732578 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaca160df_fc11_4b89_b1be_31cb87ab03e7.slice/crio-9a4502a8047ddbb6b68e8c88c2b9021bd9cc9806c9915506e8fad1a6c6abc329 WatchSource:0}: Error finding container 9a4502a8047ddbb6b68e8c88c2b9021bd9cc9806c9915506e8fad1a6c6abc329: Status 404 returned error can't find the container with id 9a4502a8047ddbb6b68e8c88c2b9021bd9cc9806c9915506e8fad1a6c6abc329 Jul 07 19:05:57.771240 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:57.771203 2568 generic.go:356] "Generic (PLEG): container finished" podID="ae611325-f231-4121-83fb-4d2e01464658" containerID="107985ff367e699a2a94eaf05b07e256e88d493f91e1b2be8d83853a3a6dbcab" exitCode=0 Jul 07 19:05:57.771644 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:57.771259 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb" event={"ID":"ae611325-f231-4121-83fb-4d2e01464658","Type":"ContainerDied","Data":"107985ff367e699a2a94eaf05b07e256e88d493f91e1b2be8d83853a3a6dbcab"} Jul 07 19:05:57.772415 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:57.772395 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bjxpv" event={"ID":"aca160df-fc11-4b89-b1be-31cb87ab03e7","Type":"ContainerStarted","Data":"9a4502a8047ddbb6b68e8c88c2b9021bd9cc9806c9915506e8fad1a6c6abc329"} Jul 07 19:05:58.932077 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:58.932029 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb" Jul 07 19:05:58.993446 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:58.993402 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbr5s\" (UniqueName: \"kubernetes.io/projected/ae611325-f231-4121-83fb-4d2e01464658-kube-api-access-kbr5s\") pod \"ae611325-f231-4121-83fb-4d2e01464658\" (UID: \"ae611325-f231-4121-83fb-4d2e01464658\") " Jul 07 19:05:58.993621 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:58.993491 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae611325-f231-4121-83fb-4d2e01464658-util\") pod \"ae611325-f231-4121-83fb-4d2e01464658\" (UID: \"ae611325-f231-4121-83fb-4d2e01464658\") " Jul 07 19:05:58.993621 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:58.993568 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae611325-f231-4121-83fb-4d2e01464658-bundle\") pod \"ae611325-f231-4121-83fb-4d2e01464658\" (UID: \"ae611325-f231-4121-83fb-4d2e01464658\") " Jul 07 19:05:58.993998 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:58.993956 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae611325-f231-4121-83fb-4d2e01464658-bundle" (OuterVolumeSpecName: "bundle") pod "ae611325-f231-4121-83fb-4d2e01464658" (UID: "ae611325-f231-4121-83fb-4d2e01464658"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:05:58.996697 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:58.996668 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae611325-f231-4121-83fb-4d2e01464658-kube-api-access-kbr5s" (OuterVolumeSpecName: "kube-api-access-kbr5s") pod "ae611325-f231-4121-83fb-4d2e01464658" (UID: "ae611325-f231-4121-83fb-4d2e01464658"). InnerVolumeSpecName "kube-api-access-kbr5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:05:59.000583 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:59.000556 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae611325-f231-4121-83fb-4d2e01464658-util" (OuterVolumeSpecName: "util") pod "ae611325-f231-4121-83fb-4d2e01464658" (UID: "ae611325-f231-4121-83fb-4d2e01464658"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:05:59.094940 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:59.094842 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbr5s\" (UniqueName: \"kubernetes.io/projected/ae611325-f231-4121-83fb-4d2e01464658-kube-api-access-kbr5s\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:05:59.094940 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:59.094875 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ae611325-f231-4121-83fb-4d2e01464658-util\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:05:59.094940 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:59.094887 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ae611325-f231-4121-83fb-4d2e01464658-bundle\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:05:59.783659 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:59.783620 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb" event={"ID":"ae611325-f231-4121-83fb-4d2e01464658","Type":"ContainerDied","Data":"787ca7090ec164624f83c9c476bc45406bb006ab9fead5f680fe1531292ccb10"} Jul 07 19:05:59.783853 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:59.783681 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="787ca7090ec164624f83c9c476bc45406bb006ab9fead5f680fe1531292ccb10" Jul 07 19:05:59.783853 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:05:59.783635 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9xtxb" Jul 07 19:06:00.789156 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:00.789122 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bjxpv" event={"ID":"aca160df-fc11-4b89-b1be-31cb87ab03e7","Type":"ContainerStarted","Data":"24a497c485c3cf7d8dd620ace3c92fc85aac342ab0c50daf2c1705027436aeb2"} Jul 07 19:06:00.807934 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:00.807887 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bjxpv" podStartSLOduration=0.934158771 podStartE2EDuration="3.807869321s" totalImagesPullingTime="2.87371055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:00.807033531 +0000 UTC m=+330.999573598" watchObservedRunningTime="2026-07-07 19:06:00.807869321 +0000 UTC m=+331.000409389" Jul 07 19:06:17.602874 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.602830 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2"] Jul 07 19:06:17.603500 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.603309 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae611325-f231-4121-83fb-4d2e01464658" containerName="util" Jul 07 19:06:17.603500 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.603332 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae611325-f231-4121-83fb-4d2e01464658" containerName="util" Jul 07 19:06:17.603500 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.603351 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae611325-f231-4121-83fb-4d2e01464658" containerName="pull" Jul 07 19:06:17.603500 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.603360 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae611325-f231-4121-83fb-4d2e01464658" containerName="pull" Jul 07 19:06:17.603500 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.603386 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae611325-f231-4121-83fb-4d2e01464658" containerName="extract" Jul 07 19:06:17.603500 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.603394 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae611325-f231-4121-83fb-4d2e01464658" containerName="extract" Jul 07 19:06:17.603500 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.603479 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae611325-f231-4121-83fb-4d2e01464658" containerName="extract" Jul 07 19:06:17.606346 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.606324 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2" Jul 07 19:06:17.608701 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.608680 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-kgw6h\"" Jul 07 19:06:17.611789 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.611768 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 07 19:06:17.611865 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.611787 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 07 19:06:17.621498 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.621474 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2"] Jul 07 19:06:17.754905 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.754874 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2\" (UID: \"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2" Jul 07 19:06:17.754905 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.754911 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56xkl\" (UniqueName: \"kubernetes.io/projected/ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64-kube-api-access-56xkl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2\" (UID: \"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2" Jul 07 19:06:17.755138 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.754942 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2\" (UID: \"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2" Jul 07 19:06:17.855964 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.855881 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2\" (UID: \"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2" Jul 07 19:06:17.855964 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.855919 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56xkl\" (UniqueName: \"kubernetes.io/projected/ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64-kube-api-access-56xkl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2\" (UID: \"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2" Jul 07 19:06:17.855964 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.855950 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2\" (UID: \"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2" Jul 07 19:06:17.856303 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.856285 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2\" (UID: \"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2" Jul 07 19:06:17.856351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.856316 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2\" (UID: \"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2" Jul 07 19:06:17.866116 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.866082 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56xkl\" (UniqueName: \"kubernetes.io/projected/ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64-kube-api-access-56xkl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2\" (UID: \"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2" Jul 07 19:06:17.915900 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:17.915863 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2" Jul 07 19:06:18.071360 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:18.071334 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2"] Jul 07 19:06:18.073192 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:06:18.073170 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff06f590_ce32_4bb5_9f4b_e8f3bd08ee64.slice/crio-1e558a1d4d4a2d19b275ee1b50818766a58bc7ceb1b9c868ad44ea141bd06abf WatchSource:0}: Error finding container 1e558a1d4d4a2d19b275ee1b50818766a58bc7ceb1b9c868ad44ea141bd06abf: Status 404 returned error can't find the container with id 1e558a1d4d4a2d19b275ee1b50818766a58bc7ceb1b9c868ad44ea141bd06abf Jul 07 19:06:18.867773 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:18.867743 2568 generic.go:356] "Generic (PLEG): container finished" podID="ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64" containerID="670bf3671d6ac9c279f2b5937d951418c0662bdb756d02bc2f3e9b5b9eb3ae2a" exitCode=0 Jul 07 19:06:18.868251 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:18.867835 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2" event={"ID":"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64","Type":"ContainerDied","Data":"670bf3671d6ac9c279f2b5937d951418c0662bdb756d02bc2f3e9b5b9eb3ae2a"} Jul 07 19:06:18.868251 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:18.867904 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2" event={"ID":"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64","Type":"ContainerStarted","Data":"1e558a1d4d4a2d19b275ee1b50818766a58bc7ceb1b9c868ad44ea141bd06abf"} Jul 07 19:06:19.875483 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:19.875451 2568 generic.go:356] "Generic (PLEG): container finished" podID="ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64" containerID="d34138035efc56925f0a33e5ae558a88f88b9e709937076e12abd4e4c50fec59" exitCode=0 Jul 07 19:06:19.875865 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:19.875521 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2" event={"ID":"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64","Type":"ContainerDied","Data":"d34138035efc56925f0a33e5ae558a88f88b9e709937076e12abd4e4c50fec59"} Jul 07 19:06:20.881896 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:20.881861 2568 generic.go:356] "Generic (PLEG): container finished" podID="ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64" containerID="624fbc2b223264e6d9c50dfed5f7fdd1c483907d4923c7df4deeaa580db735ae" exitCode=0 Jul 07 19:06:20.882375 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:20.881944 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2" event={"ID":"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64","Type":"ContainerDied","Data":"624fbc2b223264e6d9c50dfed5f7fdd1c483907d4923c7df4deeaa580db735ae"} Jul 07 19:06:22.013854 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:22.013828 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2" Jul 07 19:06:22.190761 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:22.190672 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64-bundle\") pod \"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64\" (UID: \"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64\") " Jul 07 19:06:22.190761 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:22.190739 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64-util\") pod \"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64\" (UID: \"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64\") " Jul 07 19:06:22.190939 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:22.190767 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56xkl\" (UniqueName: \"kubernetes.io/projected/ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64-kube-api-access-56xkl\") pod \"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64\" (UID: \"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64\") " Jul 07 19:06:22.191570 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:22.191538 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64-bundle" (OuterVolumeSpecName: "bundle") pod "ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64" (UID: "ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:06:22.192951 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:22.192924 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64-kube-api-access-56xkl" (OuterVolumeSpecName: "kube-api-access-56xkl") pod "ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64" (UID: "ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64"). InnerVolumeSpecName "kube-api-access-56xkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:06:22.196186 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:22.196163 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64-util" (OuterVolumeSpecName: "util") pod "ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64" (UID: "ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:06:22.291795 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:22.291712 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64-bundle\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:06:22.291795 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:22.291743 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64-util\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:06:22.291795 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:22.291754 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56xkl\" (UniqueName: \"kubernetes.io/projected/ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64-kube-api-access-56xkl\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:06:22.894036 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:22.893996 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2" event={"ID":"ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64","Type":"ContainerDied","Data":"1e558a1d4d4a2d19b275ee1b50818766a58bc7ceb1b9c868ad44ea141bd06abf"} Jul 07 19:06:22.894219 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:22.894055 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af8bmv2" Jul 07 19:06:22.894219 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:22.894065 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e558a1d4d4a2d19b275ee1b50818766a58bc7ceb1b9c868ad44ea141bd06abf" Jul 07 19:06:29.847402 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.847369 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94"] Jul 07 19:06:29.847775 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.847690 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64" containerName="pull" Jul 07 19:06:29.847775 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.847701 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64" containerName="pull" Jul 07 19:06:29.847775 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.847712 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64" containerName="extract" Jul 07 19:06:29.847775 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.847717 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64" containerName="extract" Jul 07 19:06:29.847775 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.847726 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64" containerName="util" Jul 07 19:06:29.847775 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.847732 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64" containerName="util" Jul 07 19:06:29.848010 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.847783 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff06f590-ce32-4bb5-9f4b-e8f3bd08ee64" containerName="extract" Jul 07 19:06:29.853766 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.853746 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" Jul 07 19:06:29.856261 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.856062 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 19:06:29.856261 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.856125 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 19:06:29.856261 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.856074 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 19:06:29.856261 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.856168 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 19:06:29.856261 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.856183 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4vlfr\"" Jul 07 19:06:29.858498 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.858478 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcc12345-2b14-4c96-b436-028d1a970815-tmp\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-wnf94\" (UID: \"fcc12345-2b14-4c96-b436-028d1a970815\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" Jul 07 19:06:29.858589 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.858519 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chvwq\" (UniqueName: \"kubernetes.io/projected/fcc12345-2b14-4c96-b436-028d1a970815-kube-api-access-chvwq\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-wnf94\" (UID: \"fcc12345-2b14-4c96-b436-028d1a970815\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" Jul 07 19:06:29.858589 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.858546 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fcc12345-2b14-4c96-b436-028d1a970815-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-wnf94\" (UID: \"fcc12345-2b14-4c96-b436-028d1a970815\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" Jul 07 19:06:29.858589 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.858571 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fcc12345-2b14-4c96-b436-028d1a970815-webhook-cert\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-wnf94\" (UID: \"fcc12345-2b14-4c96-b436-028d1a970815\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" Jul 07 19:06:29.858683 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.858607 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fcc12345-2b14-4c96-b436-028d1a970815-manifests\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-wnf94\" (UID: \"fcc12345-2b14-4c96-b436-028d1a970815\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" Jul 07 19:06:29.873621 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.873591 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94"] Jul 07 19:06:29.959586 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.959557 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chvwq\" (UniqueName: \"kubernetes.io/projected/fcc12345-2b14-4c96-b436-028d1a970815-kube-api-access-chvwq\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-wnf94\" (UID: \"fcc12345-2b14-4c96-b436-028d1a970815\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" Jul 07 19:06:29.959761 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.959590 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fcc12345-2b14-4c96-b436-028d1a970815-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-wnf94\" (UID: \"fcc12345-2b14-4c96-b436-028d1a970815\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" Jul 07 19:06:29.959761 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.959612 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fcc12345-2b14-4c96-b436-028d1a970815-webhook-cert\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-wnf94\" (UID: \"fcc12345-2b14-4c96-b436-028d1a970815\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" Jul 07 19:06:29.959761 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.959656 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fcc12345-2b14-4c96-b436-028d1a970815-manifests\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-wnf94\" (UID: \"fcc12345-2b14-4c96-b436-028d1a970815\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" Jul 07 19:06:29.959761 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.959709 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcc12345-2b14-4c96-b436-028d1a970815-tmp\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-wnf94\" (UID: \"fcc12345-2b14-4c96-b436-028d1a970815\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" Jul 07 19:06:29.960186 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.960158 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fcc12345-2b14-4c96-b436-028d1a970815-manifests\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-wnf94\" (UID: \"fcc12345-2b14-4c96-b436-028d1a970815\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" Jul 07 19:06:29.961930 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.961902 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcc12345-2b14-4c96-b436-028d1a970815-tmp\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-wnf94\" (UID: \"fcc12345-2b14-4c96-b436-028d1a970815\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" Jul 07 19:06:29.962233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.962212 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fcc12345-2b14-4c96-b436-028d1a970815-webhook-cert\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-wnf94\" (UID: \"fcc12345-2b14-4c96-b436-028d1a970815\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" Jul 07 19:06:29.962303 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.962233 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fcc12345-2b14-4c96-b436-028d1a970815-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-wnf94\" (UID: \"fcc12345-2b14-4c96-b436-028d1a970815\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" Jul 07 19:06:29.971851 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:29.971821 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chvwq\" (UniqueName: \"kubernetes.io/projected/fcc12345-2b14-4c96-b436-028d1a970815-kube-api-access-chvwq\") pod \"opendatahub-operator-controller-manager-54cc9bdf65-wnf94\" (UID: \"fcc12345-2b14-4c96-b436-028d1a970815\") " pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" Jul 07 19:06:30.165193 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:30.165146 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" Jul 07 19:06:30.345768 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:30.345740 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94"] Jul 07 19:06:30.348766 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:06:30.348740 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc12345_2b14_4c96_b436_028d1a970815.slice/crio-4389a83735b15efbbecf2746e9b2a8748127b353957ac5a19a316332c477937d WatchSource:0}: Error finding container 4389a83735b15efbbecf2746e9b2a8748127b353957ac5a19a316332c477937d: Status 404 returned error can't find the container with id 4389a83735b15efbbecf2746e9b2a8748127b353957ac5a19a316332c477937d Jul 07 19:06:30.932213 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:30.932169 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" event={"ID":"fcc12345-2b14-4c96-b436-028d1a970815","Type":"ContainerStarted","Data":"4389a83735b15efbbecf2746e9b2a8748127b353957ac5a19a316332c477937d"} Jul 07 19:06:33.949839 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:33.949798 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" event={"ID":"fcc12345-2b14-4c96-b436-028d1a970815","Type":"ContainerStarted","Data":"5944d54bbde5b0454305231c8b4a74f8eeaf8cdcc22527add42f194076765839"} Jul 07 19:06:34.955551 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:34.955459 2568 generic.go:356] "Generic (PLEG): container finished" podID="fcc12345-2b14-4c96-b436-028d1a970815" containerID="5944d54bbde5b0454305231c8b4a74f8eeaf8cdcc22527add42f194076765839" exitCode=0 Jul 07 19:06:34.955551 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:34.955505 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" event={"ID":"fcc12345-2b14-4c96-b436-028d1a970815","Type":"ContainerDied","Data":"5944d54bbde5b0454305231c8b4a74f8eeaf8cdcc22527add42f194076765839"} Jul 07 19:06:35.962338 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:35.962306 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" event={"ID":"fcc12345-2b14-4c96-b436-028d1a970815","Type":"ContainerStarted","Data":"3ae9a2a782366b517bb5aead6dfca97322ea1d1f43fa5ae9e410000b641dcee4"} Jul 07 19:06:35.962783 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:35.962456 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" Jul 07 19:06:36.020741 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:36.020688 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" podStartSLOduration=2.450837647 podStartE2EDuration="7.020671616s" totalImagesPullingTime="3.46962354s" totalInitContainerRuntime="1.100210429s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:36.020111979 +0000 UTC m=+366.212652066" watchObservedRunningTime="2026-07-07 19:06:36.020671616 +0000 UTC m=+366.213211760" Jul 07 19:06:46.970525 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:46.970490 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-54cc9bdf65-wnf94" Jul 07 19:06:49.788408 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:49.788367 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7cd4df5c96-dnlrc"] Jul 07 19:06:49.796615 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:49.796585 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-dnlrc" Jul 07 19:06:49.800492 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:49.800469 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 19:06:49.801098 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:49.801079 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 19:06:49.802650 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:49.802630 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 19:06:49.802756 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:49.802657 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 19:06:49.827570 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:49.827534 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cd4df5c96-dnlrc"] Jul 07 19:06:49.934931 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:49.934892 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b034d9e-3007-4ede-86ee-3478190d159e-tmp\") pod \"kube-auth-proxy-7cd4df5c96-dnlrc\" (UID: \"0b034d9e-3007-4ede-86ee-3478190d159e\") " pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-dnlrc" Jul 07 19:06:49.935136 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:49.934937 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5scn6\" (UniqueName: \"kubernetes.io/projected/0b034d9e-3007-4ede-86ee-3478190d159e-kube-api-access-5scn6\") pod \"kube-auth-proxy-7cd4df5c96-dnlrc\" (UID: \"0b034d9e-3007-4ede-86ee-3478190d159e\") " pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-dnlrc" Jul 07 19:06:49.935136 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:49.934979 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b034d9e-3007-4ede-86ee-3478190d159e-tls-certs\") pod \"kube-auth-proxy-7cd4df5c96-dnlrc\" (UID: \"0b034d9e-3007-4ede-86ee-3478190d159e\") " pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-dnlrc" Jul 07 19:06:50.036096 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.036063 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b034d9e-3007-4ede-86ee-3478190d159e-tmp\") pod \"kube-auth-proxy-7cd4df5c96-dnlrc\" (UID: \"0b034d9e-3007-4ede-86ee-3478190d159e\") " pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-dnlrc" Jul 07 19:06:50.036292 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.036104 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5scn6\" (UniqueName: \"kubernetes.io/projected/0b034d9e-3007-4ede-86ee-3478190d159e-kube-api-access-5scn6\") pod \"kube-auth-proxy-7cd4df5c96-dnlrc\" (UID: \"0b034d9e-3007-4ede-86ee-3478190d159e\") " pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-dnlrc" Jul 07 19:06:50.036292 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.036269 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b034d9e-3007-4ede-86ee-3478190d159e-tls-certs\") pod \"kube-auth-proxy-7cd4df5c96-dnlrc\" (UID: \"0b034d9e-3007-4ede-86ee-3478190d159e\") " pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-dnlrc" Jul 07 19:06:50.038509 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.038442 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b034d9e-3007-4ede-86ee-3478190d159e-tmp\") pod \"kube-auth-proxy-7cd4df5c96-dnlrc\" (UID: \"0b034d9e-3007-4ede-86ee-3478190d159e\") " pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-dnlrc" Jul 07 19:06:50.038609 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.038584 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b034d9e-3007-4ede-86ee-3478190d159e-tls-certs\") pod \"kube-auth-proxy-7cd4df5c96-dnlrc\" (UID: \"0b034d9e-3007-4ede-86ee-3478190d159e\") " pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-dnlrc" Jul 07 19:06:50.062719 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.062690 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5scn6\" (UniqueName: \"kubernetes.io/projected/0b034d9e-3007-4ede-86ee-3478190d159e-kube-api-access-5scn6\") pod \"kube-auth-proxy-7cd4df5c96-dnlrc\" (UID: \"0b034d9e-3007-4ede-86ee-3478190d159e\") " pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-dnlrc" Jul 07 19:06:50.076763 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.076728 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9kfwv"] Jul 07 19:06:50.083074 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.083037 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9kfwv" Jul 07 19:06:50.085813 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.085793 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 07 19:06:50.085922 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.085837 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-hgp5v\"" Jul 07 19:06:50.098564 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.098540 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9kfwv"] Jul 07 19:06:50.105838 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.105813 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-dnlrc" Jul 07 19:06:50.137398 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.137369 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/184e6a6c-0a82-4c5e-957d-3a344adb0d5a-cert\") pod \"odh-model-controller-7998c67dc4-9kfwv\" (UID: \"184e6a6c-0a82-4c5e-957d-3a344adb0d5a\") " pod="opendatahub/odh-model-controller-7998c67dc4-9kfwv" Jul 07 19:06:50.137559 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.137407 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mkp9\" (UniqueName: \"kubernetes.io/projected/184e6a6c-0a82-4c5e-957d-3a344adb0d5a-kube-api-access-9mkp9\") pod \"odh-model-controller-7998c67dc4-9kfwv\" (UID: \"184e6a6c-0a82-4c5e-957d-3a344adb0d5a\") " pod="opendatahub/odh-model-controller-7998c67dc4-9kfwv" Jul 07 19:06:50.238190 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.238153 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/184e6a6c-0a82-4c5e-957d-3a344adb0d5a-cert\") pod \"odh-model-controller-7998c67dc4-9kfwv\" (UID: \"184e6a6c-0a82-4c5e-957d-3a344adb0d5a\") " pod="opendatahub/odh-model-controller-7998c67dc4-9kfwv" Jul 07 19:06:50.238190 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.238195 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mkp9\" (UniqueName: \"kubernetes.io/projected/184e6a6c-0a82-4c5e-957d-3a344adb0d5a-kube-api-access-9mkp9\") pod \"odh-model-controller-7998c67dc4-9kfwv\" (UID: \"184e6a6c-0a82-4c5e-957d-3a344adb0d5a\") " pod="opendatahub/odh-model-controller-7998c67dc4-9kfwv" Jul 07 19:06:50.238392 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:06:50.238313 2568 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 07 19:06:50.238439 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:06:50.238393 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/184e6a6c-0a82-4c5e-957d-3a344adb0d5a-cert podName:184e6a6c-0a82-4c5e-957d-3a344adb0d5a nodeName:}" failed. No retries permitted until 2026-07-07 19:06:50.738372428 +0000 UTC m=+380.930912483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/184e6a6c-0a82-4c5e-957d-3a344adb0d5a-cert") pod "odh-model-controller-7998c67dc4-9kfwv" (UID: "184e6a6c-0a82-4c5e-957d-3a344adb0d5a") : secret "odh-model-controller-webhook-cert" not found Jul 07 19:06:50.254571 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.254367 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cd4df5c96-dnlrc"] Jul 07 19:06:50.256935 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:06:50.256898 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b034d9e_3007_4ede_86ee_3478190d159e.slice/crio-c8bcfc8b1d3cacd16b14c98f702ffd009954b9775789ebde0bd6832b37659960 WatchSource:0}: Error finding container c8bcfc8b1d3cacd16b14c98f702ffd009954b9775789ebde0bd6832b37659960: Status 404 returned error can't find the container with id c8bcfc8b1d3cacd16b14c98f702ffd009954b9775789ebde0bd6832b37659960 Jul 07 19:06:50.262866 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.262846 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mkp9\" (UniqueName: \"kubernetes.io/projected/184e6a6c-0a82-4c5e-957d-3a344adb0d5a-kube-api-access-9mkp9\") pod \"odh-model-controller-7998c67dc4-9kfwv\" (UID: \"184e6a6c-0a82-4c5e-957d-3a344adb0d5a\") " pod="opendatahub/odh-model-controller-7998c67dc4-9kfwv" Jul 07 19:06:50.288113 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.288080 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml"] Jul 07 19:06:50.292934 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.292874 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml" Jul 07 19:06:50.298994 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.298964 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 19:06:50.299274 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.299250 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 19:06:50.299399 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.299250 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:06:50.299399 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.299256 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 19:06:50.299693 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.299677 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fvnrv\"" Jul 07 19:06:50.305339 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.305278 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml"] Jul 07 19:06:50.306624 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.306591 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 19:06:50.439905 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.439876 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6c0c2e5b-c75b-4122-9eb8-1db975b14e53-manager-config\") pod \"lws-controller-manager-5f4bfc584b-mfmml\" (UID: \"6c0c2e5b-c75b-4122-9eb8-1db975b14e53\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml" Jul 07 19:06:50.440105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.439926 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c0c2e5b-c75b-4122-9eb8-1db975b14e53-cert\") pod \"lws-controller-manager-5f4bfc584b-mfmml\" (UID: \"6c0c2e5b-c75b-4122-9eb8-1db975b14e53\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml" Jul 07 19:06:50.440105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.439951 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c0c2e5b-c75b-4122-9eb8-1db975b14e53-metrics-cert\") pod \"lws-controller-manager-5f4bfc584b-mfmml\" (UID: \"6c0c2e5b-c75b-4122-9eb8-1db975b14e53\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml" Jul 07 19:06:50.440105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.440022 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7r58\" (UniqueName: \"kubernetes.io/projected/6c0c2e5b-c75b-4122-9eb8-1db975b14e53-kube-api-access-n7r58\") pod \"lws-controller-manager-5f4bfc584b-mfmml\" (UID: \"6c0c2e5b-c75b-4122-9eb8-1db975b14e53\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml" Jul 07 19:06:50.540911 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.540865 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c0c2e5b-c75b-4122-9eb8-1db975b14e53-cert\") pod \"lws-controller-manager-5f4bfc584b-mfmml\" (UID: \"6c0c2e5b-c75b-4122-9eb8-1db975b14e53\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml" Jul 07 19:06:50.541123 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.540920 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c0c2e5b-c75b-4122-9eb8-1db975b14e53-metrics-cert\") pod \"lws-controller-manager-5f4bfc584b-mfmml\" (UID: \"6c0c2e5b-c75b-4122-9eb8-1db975b14e53\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml" Jul 07 19:06:50.541123 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.540948 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7r58\" (UniqueName: \"kubernetes.io/projected/6c0c2e5b-c75b-4122-9eb8-1db975b14e53-kube-api-access-n7r58\") pod \"lws-controller-manager-5f4bfc584b-mfmml\" (UID: \"6c0c2e5b-c75b-4122-9eb8-1db975b14e53\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml" Jul 07 19:06:50.541123 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.541083 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6c0c2e5b-c75b-4122-9eb8-1db975b14e53-manager-config\") pod \"lws-controller-manager-5f4bfc584b-mfmml\" (UID: \"6c0c2e5b-c75b-4122-9eb8-1db975b14e53\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml" Jul 07 19:06:50.541815 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.541782 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6c0c2e5b-c75b-4122-9eb8-1db975b14e53-manager-config\") pod \"lws-controller-manager-5f4bfc584b-mfmml\" (UID: \"6c0c2e5b-c75b-4122-9eb8-1db975b14e53\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml" Jul 07 19:06:50.543449 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.543395 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c0c2e5b-c75b-4122-9eb8-1db975b14e53-metrics-cert\") pod \"lws-controller-manager-5f4bfc584b-mfmml\" (UID: \"6c0c2e5b-c75b-4122-9eb8-1db975b14e53\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml" Jul 07 19:06:50.543537 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.543446 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c0c2e5b-c75b-4122-9eb8-1db975b14e53-cert\") pod \"lws-controller-manager-5f4bfc584b-mfmml\" (UID: \"6c0c2e5b-c75b-4122-9eb8-1db975b14e53\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml" Jul 07 19:06:50.552566 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.552543 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7r58\" (UniqueName: \"kubernetes.io/projected/6c0c2e5b-c75b-4122-9eb8-1db975b14e53-kube-api-access-n7r58\") pod \"lws-controller-manager-5f4bfc584b-mfmml\" (UID: \"6c0c2e5b-c75b-4122-9eb8-1db975b14e53\") " pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml" Jul 07 19:06:50.606247 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.606210 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml" Jul 07 19:06:50.742980 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.742944 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/184e6a6c-0a82-4c5e-957d-3a344adb0d5a-cert\") pod \"odh-model-controller-7998c67dc4-9kfwv\" (UID: \"184e6a6c-0a82-4c5e-957d-3a344adb0d5a\") " pod="opendatahub/odh-model-controller-7998c67dc4-9kfwv" Jul 07 19:06:50.745286 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.745266 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/184e6a6c-0a82-4c5e-957d-3a344adb0d5a-cert\") pod \"odh-model-controller-7998c67dc4-9kfwv\" (UID: \"184e6a6c-0a82-4c5e-957d-3a344adb0d5a\") " pod="opendatahub/odh-model-controller-7998c67dc4-9kfwv" Jul 07 19:06:50.805446 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.805339 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml"] Jul 07 19:06:50.993195 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:50.993163 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9kfwv" Jul 07 19:06:51.035632 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:51.035569 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml" event={"ID":"6c0c2e5b-c75b-4122-9eb8-1db975b14e53","Type":"ContainerStarted","Data":"a238cb373c4a8dab326c506ba737bc0f0aa94d0b352165679b0f0a4732a545c2"} Jul 07 19:06:51.037291 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:51.037255 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-dnlrc" event={"ID":"0b034d9e-3007-4ede-86ee-3478190d159e","Type":"ContainerStarted","Data":"c8bcfc8b1d3cacd16b14c98f702ffd009954b9775789ebde0bd6832b37659960"} Jul 07 19:06:51.148689 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:51.148658 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9kfwv"] Jul 07 19:06:51.149497 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:06:51.149468 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184e6a6c_0a82_4c5e_957d_3a344adb0d5a.slice/crio-5f8a68a4861d5b71dd5e9f71be9281d0d1b0abd2cee23bfd5b4105985f5e0aaa WatchSource:0}: Error finding container 5f8a68a4861d5b71dd5e9f71be9281d0d1b0abd2cee23bfd5b4105985f5e0aaa: Status 404 returned error can't find the container with id 5f8a68a4861d5b71dd5e9f71be9281d0d1b0abd2cee23bfd5b4105985f5e0aaa Jul 07 19:06:52.047073 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:52.046968 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9kfwv" event={"ID":"184e6a6c-0a82-4c5e-957d-3a344adb0d5a","Type":"ContainerStarted","Data":"5f8a68a4861d5b71dd5e9f71be9281d0d1b0abd2cee23bfd5b4105985f5e0aaa"} Jul 07 19:06:53.059206 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:53.059164 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml" event={"ID":"6c0c2e5b-c75b-4122-9eb8-1db975b14e53","Type":"ContainerStarted","Data":"eac929b57f09bad3a14e69693ecf3c655cc39cb88e3aa6aca13e47ef4ace5608"} Jul 07 19:06:53.059849 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:53.059351 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml" Jul 07 19:06:53.087726 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:53.087670 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml" podStartSLOduration=1.42188023 podStartE2EDuration="3.087652047s" totalImagesPullingTime="1.665771817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:53.086509243 +0000 UTC m=+383.279049311" watchObservedRunningTime="2026-07-07 19:06:53.087652047 +0000 UTC m=+383.280192114" Jul 07 19:06:55.071213 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:55.071175 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-dnlrc" event={"ID":"0b034d9e-3007-4ede-86ee-3478190d159e","Type":"ContainerStarted","Data":"7af021fa8ad303774fcd31d6dd0e8fb64f0ed9f655c1f0a192a7031711ebbee0"} Jul 07 19:06:55.072819 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:55.072774 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9kfwv" event={"ID":"184e6a6c-0a82-4c5e-957d-3a344adb0d5a","Type":"ContainerStarted","Data":"bba0bad3535c7299361c19ee47a0811589499354454bfe83cf2972d2552045a4"} Jul 07 19:06:55.072926 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:55.072837 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-9kfwv" Jul 07 19:06:55.092708 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:55.092610 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7cd4df5c96-dnlrc" podStartSLOduration=1.522667795 podStartE2EDuration="6.092594099s" totalImagesPullingTime="4.569926304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:55.092032949 +0000 UTC m=+385.284573017" watchObservedRunningTime="2026-07-07 19:06:55.092594099 +0000 UTC m=+385.285134167" Jul 07 19:06:55.115996 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:06:55.115933 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-9kfwv" podStartSLOduration=1.433302083 podStartE2EDuration="5.115908962s" totalImagesPullingTime="3.682606879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:55.11369488 +0000 UTC m=+385.306234951" watchObservedRunningTime="2026-07-07 19:06:55.115908962 +0000 UTC m=+385.308449029" Jul 07 19:07:04.066857 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:07:04.066824 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5f4bfc584b-mfmml" Jul 07 19:07:06.080306 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:07:06.080274 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-9kfwv" Jul 07 19:08:06.222141 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.222061 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6"] Jul 07 19:08:06.224922 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.224905 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6" Jul 07 19:08:06.227415 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.227386 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 19:08:06.228040 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.228021 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 19:08:06.228157 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.228081 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-f2sxg\"" Jul 07 19:08:06.233840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.233821 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6"] Jul 07 19:08:06.314455 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.314405 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b1b73ff-28a6-4e52-85e0-df7d72f95cb7-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6\" (UID: \"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6" Jul 07 19:08:06.314455 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.314456 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b1b73ff-28a6-4e52-85e0-df7d72f95cb7-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6\" (UID: \"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6" Jul 07 19:08:06.314668 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.314536 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctbl7\" (UniqueName: \"kubernetes.io/projected/8b1b73ff-28a6-4e52-85e0-df7d72f95cb7-kube-api-access-ctbl7\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6\" (UID: \"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6" Jul 07 19:08:06.415172 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.415135 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctbl7\" (UniqueName: \"kubernetes.io/projected/8b1b73ff-28a6-4e52-85e0-df7d72f95cb7-kube-api-access-ctbl7\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6\" (UID: \"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6" Jul 07 19:08:06.415321 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.415207 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b1b73ff-28a6-4e52-85e0-df7d72f95cb7-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6\" (UID: \"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6" Jul 07 19:08:06.415321 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.415240 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b1b73ff-28a6-4e52-85e0-df7d72f95cb7-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6\" (UID: \"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6" Jul 07 19:08:06.415659 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.415643 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b1b73ff-28a6-4e52-85e0-df7d72f95cb7-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6\" (UID: \"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6" Jul 07 19:08:06.415695 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.415667 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b1b73ff-28a6-4e52-85e0-df7d72f95cb7-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6\" (UID: \"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6" Jul 07 19:08:06.424727 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.424693 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctbl7\" (UniqueName: \"kubernetes.io/projected/8b1b73ff-28a6-4e52-85e0-df7d72f95cb7-kube-api-access-ctbl7\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6\" (UID: \"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6" Jul 07 19:08:06.535528 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.535441 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6" Jul 07 19:08:06.670374 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.670341 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6"] Jul 07 19:08:06.673079 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:08:06.673022 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1b73ff_28a6_4e52_85e0_df7d72f95cb7.slice/crio-a679d8e7fd9f018428a3be247aac276573dbd4954fd1bf63a0e752618094da21 WatchSource:0}: Error finding container a679d8e7fd9f018428a3be247aac276573dbd4954fd1bf63a0e752618094da21: Status 404 returned error can't find the container with id a679d8e7fd9f018428a3be247aac276573dbd4954fd1bf63a0e752618094da21 Jul 07 19:08:06.818372 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.818293 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6"] Jul 07 19:08:06.820997 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.820981 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6" Jul 07 19:08:06.831330 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.831305 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6"] Jul 07 19:08:06.920673 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.920638 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prxhj\" (UniqueName: \"kubernetes.io/projected/6a2cbdd2-093d-495d-af87-de9ea1ec7e45-kube-api-access-prxhj\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6\" (UID: \"6a2cbdd2-093d-495d-af87-de9ea1ec7e45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6" Jul 07 19:08:06.920863 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.920690 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a2cbdd2-093d-495d-af87-de9ea1ec7e45-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6\" (UID: \"6a2cbdd2-093d-495d-af87-de9ea1ec7e45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6" Jul 07 19:08:06.920863 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:06.920734 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a2cbdd2-093d-495d-af87-de9ea1ec7e45-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6\" (UID: \"6a2cbdd2-093d-495d-af87-de9ea1ec7e45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6" Jul 07 19:08:07.021504 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.021464 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prxhj\" (UniqueName: \"kubernetes.io/projected/6a2cbdd2-093d-495d-af87-de9ea1ec7e45-kube-api-access-prxhj\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6\" (UID: \"6a2cbdd2-093d-495d-af87-de9ea1ec7e45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6" Jul 07 19:08:07.021676 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.021526 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a2cbdd2-093d-495d-af87-de9ea1ec7e45-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6\" (UID: \"6a2cbdd2-093d-495d-af87-de9ea1ec7e45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6" Jul 07 19:08:07.021676 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.021572 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a2cbdd2-093d-495d-af87-de9ea1ec7e45-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6\" (UID: \"6a2cbdd2-093d-495d-af87-de9ea1ec7e45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6" Jul 07 19:08:07.021876 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.021855 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a2cbdd2-093d-495d-af87-de9ea1ec7e45-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6\" (UID: \"6a2cbdd2-093d-495d-af87-de9ea1ec7e45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6" Jul 07 19:08:07.021945 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.021912 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a2cbdd2-093d-495d-af87-de9ea1ec7e45-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6\" (UID: \"6a2cbdd2-093d-495d-af87-de9ea1ec7e45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6" Jul 07 19:08:07.031384 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.031351 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prxhj\" (UniqueName: \"kubernetes.io/projected/6a2cbdd2-093d-495d-af87-de9ea1ec7e45-kube-api-access-prxhj\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6\" (UID: \"6a2cbdd2-093d-495d-af87-de9ea1ec7e45\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6" Jul 07 19:08:07.144504 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.144470 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6" Jul 07 19:08:07.252523 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.252484 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w"] Jul 07 19:08:07.256308 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.256280 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w" Jul 07 19:08:07.267686 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.267660 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w"] Jul 07 19:08:07.283076 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.283035 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6"] Jul 07 19:08:07.285490 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:08:07.285465 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2cbdd2_093d_495d_af87_de9ea1ec7e45.slice/crio-a59ad5c7ceeb5d56a4be6403a9b5b872cb7584a212800b6e65ecd59564e309a5 WatchSource:0}: Error finding container a59ad5c7ceeb5d56a4be6403a9b5b872cb7584a212800b6e65ecd59564e309a5: Status 404 returned error can't find the container with id a59ad5c7ceeb5d56a4be6403a9b5b872cb7584a212800b6e65ecd59564e309a5 Jul 07 19:08:07.323545 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.323518 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c8b8b63-d0c6-437a-b0db-525553f775d0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w\" (UID: \"6c8b8b63-d0c6-437a-b0db-525553f775d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w" Jul 07 19:08:07.323655 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.323562 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2fsl\" (UniqueName: \"kubernetes.io/projected/6c8b8b63-d0c6-437a-b0db-525553f775d0-kube-api-access-m2fsl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w\" (UID: \"6c8b8b63-d0c6-437a-b0db-525553f775d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w" Jul 07 19:08:07.323728 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.323703 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c8b8b63-d0c6-437a-b0db-525553f775d0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w\" (UID: \"6c8b8b63-d0c6-437a-b0db-525553f775d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w" Jul 07 19:08:07.415529 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.415453 2568 generic.go:356] "Generic (PLEG): container finished" podID="8b1b73ff-28a6-4e52-85e0-df7d72f95cb7" containerID="78940ffee4506cb16022b7591505ca7ba47928d61f94dbc9c4aabc5ab375c675" exitCode=0 Jul 07 19:08:07.415685 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.415540 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6" event={"ID":"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7","Type":"ContainerDied","Data":"78940ffee4506cb16022b7591505ca7ba47928d61f94dbc9c4aabc5ab375c675"} Jul 07 19:08:07.415685 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.415594 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6" event={"ID":"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7","Type":"ContainerStarted","Data":"a679d8e7fd9f018428a3be247aac276573dbd4954fd1bf63a0e752618094da21"} Jul 07 19:08:07.417156 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.417134 2568 generic.go:356] "Generic (PLEG): container finished" podID="6a2cbdd2-093d-495d-af87-de9ea1ec7e45" containerID="30c24a3ff1e10ae5fb40180c5f3c2d434b5b1eb5777de49c1f9d8568b82056e8" exitCode=0 Jul 07 19:08:07.417249 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.417200 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6" event={"ID":"6a2cbdd2-093d-495d-af87-de9ea1ec7e45","Type":"ContainerDied","Data":"30c24a3ff1e10ae5fb40180c5f3c2d434b5b1eb5777de49c1f9d8568b82056e8"} Jul 07 19:08:07.417249 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.417229 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6" event={"ID":"6a2cbdd2-093d-495d-af87-de9ea1ec7e45","Type":"ContainerStarted","Data":"a59ad5c7ceeb5d56a4be6403a9b5b872cb7584a212800b6e65ecd59564e309a5"} Jul 07 19:08:07.424308 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.424288 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c8b8b63-d0c6-437a-b0db-525553f775d0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w\" (UID: \"6c8b8b63-d0c6-437a-b0db-525553f775d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w" Jul 07 19:08:07.424408 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.424327 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2fsl\" (UniqueName: \"kubernetes.io/projected/6c8b8b63-d0c6-437a-b0db-525553f775d0-kube-api-access-m2fsl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w\" (UID: \"6c8b8b63-d0c6-437a-b0db-525553f775d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w" Jul 07 19:08:07.424463 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.424416 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c8b8b63-d0c6-437a-b0db-525553f775d0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w\" (UID: \"6c8b8b63-d0c6-437a-b0db-525553f775d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w" Jul 07 19:08:07.424638 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.424619 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c8b8b63-d0c6-437a-b0db-525553f775d0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w\" (UID: \"6c8b8b63-d0c6-437a-b0db-525553f775d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w" Jul 07 19:08:07.424697 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.424681 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c8b8b63-d0c6-437a-b0db-525553f775d0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w\" (UID: \"6c8b8b63-d0c6-437a-b0db-525553f775d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w" Jul 07 19:08:07.434659 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.434637 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2fsl\" (UniqueName: \"kubernetes.io/projected/6c8b8b63-d0c6-437a-b0db-525553f775d0-kube-api-access-m2fsl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w\" (UID: \"6c8b8b63-d0c6-437a-b0db-525553f775d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w" Jul 07 19:08:07.567364 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.567329 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w" Jul 07 19:08:07.625111 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.625075 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd"] Jul 07 19:08:07.629067 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.629021 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd" Jul 07 19:08:07.643011 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.642985 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd"] Jul 07 19:08:07.725559 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.725534 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w"] Jul 07 19:08:07.727597 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.727575 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/384eb23f-f99e-4d93-b0f2-47ba2b356d99-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd\" (UID: \"384eb23f-f99e-4d93-b0f2-47ba2b356d99\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd" Jul 07 19:08:07.727721 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:08:07.727590 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8b8b63_d0c6_437a_b0db_525553f775d0.slice/crio-2390f5544c26f57427cd1b8ad0ecffbd954224af811af12e98695c788f3d8b59 WatchSource:0}: Error finding container 2390f5544c26f57427cd1b8ad0ecffbd954224af811af12e98695c788f3d8b59: Status 404 returned error can't find the container with id 2390f5544c26f57427cd1b8ad0ecffbd954224af811af12e98695c788f3d8b59 Jul 07 19:08:07.727721 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.727622 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zngjn\" (UniqueName: \"kubernetes.io/projected/384eb23f-f99e-4d93-b0f2-47ba2b356d99-kube-api-access-zngjn\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd\" (UID: \"384eb23f-f99e-4d93-b0f2-47ba2b356d99\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd" Jul 07 19:08:07.727721 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.727649 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/384eb23f-f99e-4d93-b0f2-47ba2b356d99-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd\" (UID: \"384eb23f-f99e-4d93-b0f2-47ba2b356d99\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd" Jul 07 19:08:07.828251 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.828221 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/384eb23f-f99e-4d93-b0f2-47ba2b356d99-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd\" (UID: \"384eb23f-f99e-4d93-b0f2-47ba2b356d99\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd" Jul 07 19:08:07.828406 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.828280 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zngjn\" (UniqueName: \"kubernetes.io/projected/384eb23f-f99e-4d93-b0f2-47ba2b356d99-kube-api-access-zngjn\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd\" (UID: \"384eb23f-f99e-4d93-b0f2-47ba2b356d99\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd" Jul 07 19:08:07.828406 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.828310 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/384eb23f-f99e-4d93-b0f2-47ba2b356d99-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd\" (UID: \"384eb23f-f99e-4d93-b0f2-47ba2b356d99\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd" Jul 07 19:08:07.828595 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.828578 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/384eb23f-f99e-4d93-b0f2-47ba2b356d99-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd\" (UID: \"384eb23f-f99e-4d93-b0f2-47ba2b356d99\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd" Jul 07 19:08:07.828671 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.828653 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/384eb23f-f99e-4d93-b0f2-47ba2b356d99-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd\" (UID: \"384eb23f-f99e-4d93-b0f2-47ba2b356d99\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd" Jul 07 19:08:07.839491 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.839467 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zngjn\" (UniqueName: \"kubernetes.io/projected/384eb23f-f99e-4d93-b0f2-47ba2b356d99-kube-api-access-zngjn\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd\" (UID: \"384eb23f-f99e-4d93-b0f2-47ba2b356d99\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd" Jul 07 19:08:07.938828 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:07.938794 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd" Jul 07 19:08:08.108367 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:08.108331 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd"] Jul 07 19:08:08.127531 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:08:08.127497 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod384eb23f_f99e_4d93_b0f2_47ba2b356d99.slice/crio-ec4320ec732a899a8071b45c3377c95264851bc72dbfe03d46ba42e978158d32 WatchSource:0}: Error finding container ec4320ec732a899a8071b45c3377c95264851bc72dbfe03d46ba42e978158d32: Status 404 returned error can't find the container with id ec4320ec732a899a8071b45c3377c95264851bc72dbfe03d46ba42e978158d32 Jul 07 19:08:08.424204 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:08.424168 2568 generic.go:356] "Generic (PLEG): container finished" podID="384eb23f-f99e-4d93-b0f2-47ba2b356d99" containerID="d47681d3bd3c96a241bd8d97c36510eeee0669fa98bd9ac5679107c9f9b23c62" exitCode=0 Jul 07 19:08:08.424617 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:08.424255 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd" event={"ID":"384eb23f-f99e-4d93-b0f2-47ba2b356d99","Type":"ContainerDied","Data":"d47681d3bd3c96a241bd8d97c36510eeee0669fa98bd9ac5679107c9f9b23c62"} Jul 07 19:08:08.424617 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:08.424306 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd" event={"ID":"384eb23f-f99e-4d93-b0f2-47ba2b356d99","Type":"ContainerStarted","Data":"ec4320ec732a899a8071b45c3377c95264851bc72dbfe03d46ba42e978158d32"} Jul 07 19:08:08.426275 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:08.426253 2568 generic.go:356] "Generic (PLEG): container finished" podID="8b1b73ff-28a6-4e52-85e0-df7d72f95cb7" containerID="10eecaba2771baa5b57b1b4a685944a1802014bba7605619578e2c70940bd754" exitCode=0 Jul 07 19:08:08.426387 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:08.426306 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6" event={"ID":"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7","Type":"ContainerDied","Data":"10eecaba2771baa5b57b1b4a685944a1802014bba7605619578e2c70940bd754"} Jul 07 19:08:08.428264 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:08.428242 2568 generic.go:356] "Generic (PLEG): container finished" podID="6a2cbdd2-093d-495d-af87-de9ea1ec7e45" containerID="6f26b8f89083dc4a59218b9405f2f13d3fad68208bdb37918e586d09144c17ee" exitCode=0 Jul 07 19:08:08.428349 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:08.428323 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6" event={"ID":"6a2cbdd2-093d-495d-af87-de9ea1ec7e45","Type":"ContainerDied","Data":"6f26b8f89083dc4a59218b9405f2f13d3fad68208bdb37918e586d09144c17ee"} Jul 07 19:08:08.429978 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:08.429956 2568 generic.go:356] "Generic (PLEG): container finished" podID="6c8b8b63-d0c6-437a-b0db-525553f775d0" containerID="79a5f4e5e0c32df0ac9fae57aa31581c70cd8c7ee42b01144dd6ba62209a028e" exitCode=0 Jul 07 19:08:08.430097 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:08.430000 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w" event={"ID":"6c8b8b63-d0c6-437a-b0db-525553f775d0","Type":"ContainerDied","Data":"79a5f4e5e0c32df0ac9fae57aa31581c70cd8c7ee42b01144dd6ba62209a028e"} Jul 07 19:08:08.430097 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:08.430033 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w" event={"ID":"6c8b8b63-d0c6-437a-b0db-525553f775d0","Type":"ContainerStarted","Data":"2390f5544c26f57427cd1b8ad0ecffbd954224af811af12e98695c788f3d8b59"} Jul 07 19:08:09.437592 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:09.437555 2568 generic.go:356] "Generic (PLEG): container finished" podID="8b1b73ff-28a6-4e52-85e0-df7d72f95cb7" containerID="3286ed580110f91c647bd714f2f88619f23c9da5b975acee44f7aac1fe6c705b" exitCode=0 Jul 07 19:08:09.437974 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:09.437639 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6" event={"ID":"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7","Type":"ContainerDied","Data":"3286ed580110f91c647bd714f2f88619f23c9da5b975acee44f7aac1fe6c705b"} Jul 07 19:08:09.439734 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:09.439708 2568 generic.go:356] "Generic (PLEG): container finished" podID="6a2cbdd2-093d-495d-af87-de9ea1ec7e45" containerID="2e5be5e9026f9989ef644b3fe59f6e0abaec4cec0a742faceeddbe66979c6ffe" exitCode=0 Jul 07 19:08:09.439874 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:09.439789 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6" event={"ID":"6a2cbdd2-093d-495d-af87-de9ea1ec7e45","Type":"ContainerDied","Data":"2e5be5e9026f9989ef644b3fe59f6e0abaec4cec0a742faceeddbe66979c6ffe"} Jul 07 19:08:09.441560 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:09.441539 2568 generic.go:356] "Generic (PLEG): container finished" podID="6c8b8b63-d0c6-437a-b0db-525553f775d0" containerID="cdec9fcdc9514e53e8af2bb9280c168ad930e2731373890d7b04518c939665b3" exitCode=0 Jul 07 19:08:09.441676 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:09.441635 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w" event={"ID":"6c8b8b63-d0c6-437a-b0db-525553f775d0","Type":"ContainerDied","Data":"cdec9fcdc9514e53e8af2bb9280c168ad930e2731373890d7b04518c939665b3"} Jul 07 19:08:09.443495 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:09.443477 2568 generic.go:356] "Generic (PLEG): container finished" podID="384eb23f-f99e-4d93-b0f2-47ba2b356d99" containerID="2340c13c8e296f7a7f2ff1c96049b2d1324e02eb669b7ac384fea252e656487a" exitCode=0 Jul 07 19:08:09.443570 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:09.443516 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd" event={"ID":"384eb23f-f99e-4d93-b0f2-47ba2b356d99","Type":"ContainerDied","Data":"2340c13c8e296f7a7f2ff1c96049b2d1324e02eb669b7ac384fea252e656487a"} Jul 07 19:08:10.450287 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.450257 2568 generic.go:356] "Generic (PLEG): container finished" podID="384eb23f-f99e-4d93-b0f2-47ba2b356d99" containerID="c69664d62c3059054bbb57e8dd18a224e91f23467be2faaa693f8ae398541c9a" exitCode=0 Jul 07 19:08:10.450706 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.450337 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd" event={"ID":"384eb23f-f99e-4d93-b0f2-47ba2b356d99","Type":"ContainerDied","Data":"c69664d62c3059054bbb57e8dd18a224e91f23467be2faaa693f8ae398541c9a"} Jul 07 19:08:10.452390 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.452362 2568 generic.go:356] "Generic (PLEG): container finished" podID="6c8b8b63-d0c6-437a-b0db-525553f775d0" containerID="0082be316fb5892c24bc3613a81103af458b01edd4cfa615db8099e65aa23546" exitCode=0 Jul 07 19:08:10.452486 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.452437 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w" event={"ID":"6c8b8b63-d0c6-437a-b0db-525553f775d0","Type":"ContainerDied","Data":"0082be316fb5892c24bc3613a81103af458b01edd4cfa615db8099e65aa23546"} Jul 07 19:08:10.589241 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.589216 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6" Jul 07 19:08:10.626672 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.626647 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6" Jul 07 19:08:10.657305 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.657279 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a2cbdd2-093d-495d-af87-de9ea1ec7e45-util\") pod \"6a2cbdd2-093d-495d-af87-de9ea1ec7e45\" (UID: \"6a2cbdd2-093d-495d-af87-de9ea1ec7e45\") " Jul 07 19:08:10.657472 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.657329 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a2cbdd2-093d-495d-af87-de9ea1ec7e45-bundle\") pod \"6a2cbdd2-093d-495d-af87-de9ea1ec7e45\" (UID: \"6a2cbdd2-093d-495d-af87-de9ea1ec7e45\") " Jul 07 19:08:10.657472 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.657381 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prxhj\" (UniqueName: \"kubernetes.io/projected/6a2cbdd2-093d-495d-af87-de9ea1ec7e45-kube-api-access-prxhj\") pod \"6a2cbdd2-093d-495d-af87-de9ea1ec7e45\" (UID: \"6a2cbdd2-093d-495d-af87-de9ea1ec7e45\") " Jul 07 19:08:10.657472 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.657414 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b1b73ff-28a6-4e52-85e0-df7d72f95cb7-util\") pod \"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7\" (UID: \"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7\") " Jul 07 19:08:10.657639 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.657485 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b1b73ff-28a6-4e52-85e0-df7d72f95cb7-bundle\") pod \"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7\" (UID: \"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7\") " Jul 07 19:08:10.657639 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.657528 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctbl7\" (UniqueName: \"kubernetes.io/projected/8b1b73ff-28a6-4e52-85e0-df7d72f95cb7-kube-api-access-ctbl7\") pod \"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7\" (UID: \"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7\") " Jul 07 19:08:10.657938 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.657902 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2cbdd2-093d-495d-af87-de9ea1ec7e45-bundle" (OuterVolumeSpecName: "bundle") pod "6a2cbdd2-093d-495d-af87-de9ea1ec7e45" (UID: "6a2cbdd2-093d-495d-af87-de9ea1ec7e45"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:08:10.658166 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.658139 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a2cbdd2-093d-495d-af87-de9ea1ec7e45-bundle\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:08:10.658302 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.658161 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1b73ff-28a6-4e52-85e0-df7d72f95cb7-bundle" (OuterVolumeSpecName: "bundle") pod "8b1b73ff-28a6-4e52-85e0-df7d72f95cb7" (UID: "8b1b73ff-28a6-4e52-85e0-df7d72f95cb7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:08:10.659805 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.659781 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a2cbdd2-093d-495d-af87-de9ea1ec7e45-kube-api-access-prxhj" (OuterVolumeSpecName: "kube-api-access-prxhj") pod "6a2cbdd2-093d-495d-af87-de9ea1ec7e45" (UID: "6a2cbdd2-093d-495d-af87-de9ea1ec7e45"). InnerVolumeSpecName "kube-api-access-prxhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:08:10.660281 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.660254 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1b73ff-28a6-4e52-85e0-df7d72f95cb7-kube-api-access-ctbl7" (OuterVolumeSpecName: "kube-api-access-ctbl7") pod "8b1b73ff-28a6-4e52-85e0-df7d72f95cb7" (UID: "8b1b73ff-28a6-4e52-85e0-df7d72f95cb7"). InnerVolumeSpecName "kube-api-access-ctbl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:08:10.664409 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.664380 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a2cbdd2-093d-495d-af87-de9ea1ec7e45-util" (OuterVolumeSpecName: "util") pod "6a2cbdd2-093d-495d-af87-de9ea1ec7e45" (UID: "6a2cbdd2-093d-495d-af87-de9ea1ec7e45"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:08:10.664821 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.664795 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1b73ff-28a6-4e52-85e0-df7d72f95cb7-util" (OuterVolumeSpecName: "util") pod "8b1b73ff-28a6-4e52-85e0-df7d72f95cb7" (UID: "8b1b73ff-28a6-4e52-85e0-df7d72f95cb7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:08:10.759115 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.758996 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b1b73ff-28a6-4e52-85e0-df7d72f95cb7-bundle\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:08:10.759115 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.759035 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctbl7\" (UniqueName: \"kubernetes.io/projected/8b1b73ff-28a6-4e52-85e0-df7d72f95cb7-kube-api-access-ctbl7\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:08:10.759115 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.759069 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a2cbdd2-093d-495d-af87-de9ea1ec7e45-util\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:08:10.759115 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.759078 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prxhj\" (UniqueName: \"kubernetes.io/projected/6a2cbdd2-093d-495d-af87-de9ea1ec7e45-kube-api-access-prxhj\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:08:10.759115 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:10.759086 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b1b73ff-28a6-4e52-85e0-df7d72f95cb7-util\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:08:11.458997 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.458967 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6" Jul 07 19:08:11.459470 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.459000 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6" event={"ID":"8b1b73ff-28a6-4e52-85e0-df7d72f95cb7","Type":"ContainerDied","Data":"a679d8e7fd9f018428a3be247aac276573dbd4954fd1bf63a0e752618094da21"} Jul 07 19:08:11.459470 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.459068 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a679d8e7fd9f018428a3be247aac276573dbd4954fd1bf63a0e752618094da21" Jul 07 19:08:11.460935 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.460903 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6" event={"ID":"6a2cbdd2-093d-495d-af87-de9ea1ec7e45","Type":"ContainerDied","Data":"a59ad5c7ceeb5d56a4be6403a9b5b872cb7584a212800b6e65ecd59564e309a5"} Jul 07 19:08:11.461069 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.460960 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a59ad5c7ceeb5d56a4be6403a9b5b872cb7584a212800b6e65ecd59564e309a5" Jul 07 19:08:11.461069 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.460916 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6" Jul 07 19:08:11.622274 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.622253 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd" Jul 07 19:08:11.624610 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.624592 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w" Jul 07 19:08:11.665381 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.665349 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/384eb23f-f99e-4d93-b0f2-47ba2b356d99-bundle\") pod \"384eb23f-f99e-4d93-b0f2-47ba2b356d99\" (UID: \"384eb23f-f99e-4d93-b0f2-47ba2b356d99\") " Jul 07 19:08:11.665623 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.665443 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zngjn\" (UniqueName: \"kubernetes.io/projected/384eb23f-f99e-4d93-b0f2-47ba2b356d99-kube-api-access-zngjn\") pod \"384eb23f-f99e-4d93-b0f2-47ba2b356d99\" (UID: \"384eb23f-f99e-4d93-b0f2-47ba2b356d99\") " Jul 07 19:08:11.665623 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.665477 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2fsl\" (UniqueName: \"kubernetes.io/projected/6c8b8b63-d0c6-437a-b0db-525553f775d0-kube-api-access-m2fsl\") pod \"6c8b8b63-d0c6-437a-b0db-525553f775d0\" (UID: \"6c8b8b63-d0c6-437a-b0db-525553f775d0\") " Jul 07 19:08:11.665623 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.665502 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/384eb23f-f99e-4d93-b0f2-47ba2b356d99-util\") pod \"384eb23f-f99e-4d93-b0f2-47ba2b356d99\" (UID: \"384eb23f-f99e-4d93-b0f2-47ba2b356d99\") " Jul 07 19:08:11.665623 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.665568 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c8b8b63-d0c6-437a-b0db-525553f775d0-bundle\") pod \"6c8b8b63-d0c6-437a-b0db-525553f775d0\" (UID: \"6c8b8b63-d0c6-437a-b0db-525553f775d0\") " Jul 07 19:08:11.665880 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.665625 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c8b8b63-d0c6-437a-b0db-525553f775d0-util\") pod \"6c8b8b63-d0c6-437a-b0db-525553f775d0\" (UID: \"6c8b8b63-d0c6-437a-b0db-525553f775d0\") " Jul 07 19:08:11.666964 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.666820 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384eb23f-f99e-4d93-b0f2-47ba2b356d99-bundle" (OuterVolumeSpecName: "bundle") pod "384eb23f-f99e-4d93-b0f2-47ba2b356d99" (UID: "384eb23f-f99e-4d93-b0f2-47ba2b356d99"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:08:11.666964 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.666922 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8b8b63-d0c6-437a-b0db-525553f775d0-bundle" (OuterVolumeSpecName: "bundle") pod "6c8b8b63-d0c6-437a-b0db-525553f775d0" (UID: "6c8b8b63-d0c6-437a-b0db-525553f775d0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:08:11.668936 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.668861 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8b8b63-d0c6-437a-b0db-525553f775d0-kube-api-access-m2fsl" (OuterVolumeSpecName: "kube-api-access-m2fsl") pod "6c8b8b63-d0c6-437a-b0db-525553f775d0" (UID: "6c8b8b63-d0c6-437a-b0db-525553f775d0"). InnerVolumeSpecName "kube-api-access-m2fsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:08:11.669551 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.669518 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384eb23f-f99e-4d93-b0f2-47ba2b356d99-kube-api-access-zngjn" (OuterVolumeSpecName: "kube-api-access-zngjn") pod "384eb23f-f99e-4d93-b0f2-47ba2b356d99" (UID: "384eb23f-f99e-4d93-b0f2-47ba2b356d99"). InnerVolumeSpecName "kube-api-access-zngjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:08:11.672259 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.672228 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/384eb23f-f99e-4d93-b0f2-47ba2b356d99-util" (OuterVolumeSpecName: "util") pod "384eb23f-f99e-4d93-b0f2-47ba2b356d99" (UID: "384eb23f-f99e-4d93-b0f2-47ba2b356d99"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:08:11.674014 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.673989 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8b8b63-d0c6-437a-b0db-525553f775d0-util" (OuterVolumeSpecName: "util") pod "6c8b8b63-d0c6-437a-b0db-525553f775d0" (UID: "6c8b8b63-d0c6-437a-b0db-525553f775d0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:08:11.766591 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.766498 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6c8b8b63-d0c6-437a-b0db-525553f775d0-bundle\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:08:11.766591 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.766536 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6c8b8b63-d0c6-437a-b0db-525553f775d0-util\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:08:11.766591 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.766546 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/384eb23f-f99e-4d93-b0f2-47ba2b356d99-bundle\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:08:11.766591 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.766555 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zngjn\" (UniqueName: \"kubernetes.io/projected/384eb23f-f99e-4d93-b0f2-47ba2b356d99-kube-api-access-zngjn\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:08:11.766591 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.766566 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2fsl\" (UniqueName: \"kubernetes.io/projected/6c8b8b63-d0c6-437a-b0db-525553f775d0-kube-api-access-m2fsl\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:08:11.766591 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:11.766574 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/384eb23f-f99e-4d93-b0f2-47ba2b356d99-util\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:08:12.467316 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:12.467285 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w" Jul 07 19:08:12.467724 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:12.467311 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w" event={"ID":"6c8b8b63-d0c6-437a-b0db-525553f775d0","Type":"ContainerDied","Data":"2390f5544c26f57427cd1b8ad0ecffbd954224af811af12e98695c788f3d8b59"} Jul 07 19:08:12.467724 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:12.467368 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2390f5544c26f57427cd1b8ad0ecffbd954224af811af12e98695c788f3d8b59" Jul 07 19:08:12.469409 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:12.469351 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd" event={"ID":"384eb23f-f99e-4d93-b0f2-47ba2b356d99","Type":"ContainerDied","Data":"ec4320ec732a899a8071b45c3377c95264851bc72dbfe03d46ba42e978158d32"} Jul 07 19:08:12.469409 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:12.469381 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd" Jul 07 19:08:12.469409 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:12.469395 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec4320ec732a899a8071b45c3377c95264851bc72dbfe03d46ba42e978158d32" Jul 07 19:08:31.544301 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544256 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-wv44m"] Jul 07 19:08:31.544758 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544739 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c8b8b63-d0c6-437a-b0db-525553f775d0" containerName="util" Jul 07 19:08:31.544825 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544763 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c8b8b63-d0c6-437a-b0db-525553f775d0" containerName="util" Jul 07 19:08:31.544825 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544786 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b1b73ff-28a6-4e52-85e0-df7d72f95cb7" containerName="pull" Jul 07 19:08:31.544825 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544794 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b1b73ff-28a6-4e52-85e0-df7d72f95cb7" containerName="pull" Jul 07 19:08:31.544825 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544806 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c8b8b63-d0c6-437a-b0db-525553f775d0" containerName="extract" Jul 07 19:08:31.544825 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544815 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c8b8b63-d0c6-437a-b0db-525553f775d0" containerName="extract" Jul 07 19:08:31.545000 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544828 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c8b8b63-d0c6-437a-b0db-525553f775d0" containerName="pull" Jul 07 19:08:31.545000 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544836 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c8b8b63-d0c6-437a-b0db-525553f775d0" containerName="pull" Jul 07 19:08:31.545000 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544850 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="384eb23f-f99e-4d93-b0f2-47ba2b356d99" containerName="util" Jul 07 19:08:31.545000 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544858 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="384eb23f-f99e-4d93-b0f2-47ba2b356d99" containerName="util" Jul 07 19:08:31.545000 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544867 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="384eb23f-f99e-4d93-b0f2-47ba2b356d99" containerName="pull" Jul 07 19:08:31.545000 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544876 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="384eb23f-f99e-4d93-b0f2-47ba2b356d99" containerName="pull" Jul 07 19:08:31.545000 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544893 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b1b73ff-28a6-4e52-85e0-df7d72f95cb7" containerName="util" Jul 07 19:08:31.545000 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544900 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b1b73ff-28a6-4e52-85e0-df7d72f95cb7" containerName="util" Jul 07 19:08:31.545000 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544912 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b1b73ff-28a6-4e52-85e0-df7d72f95cb7" containerName="extract" Jul 07 19:08:31.545000 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544920 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b1b73ff-28a6-4e52-85e0-df7d72f95cb7" containerName="extract" Jul 07 19:08:31.545000 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544937 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a2cbdd2-093d-495d-af87-de9ea1ec7e45" containerName="pull" Jul 07 19:08:31.545000 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544945 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a2cbdd2-093d-495d-af87-de9ea1ec7e45" containerName="pull" Jul 07 19:08:31.545000 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544953 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a2cbdd2-093d-495d-af87-de9ea1ec7e45" containerName="extract" Jul 07 19:08:31.545000 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544962 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a2cbdd2-093d-495d-af87-de9ea1ec7e45" containerName="extract" Jul 07 19:08:31.545000 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544976 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="384eb23f-f99e-4d93-b0f2-47ba2b356d99" containerName="extract" Jul 07 19:08:31.545000 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.544984 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="384eb23f-f99e-4d93-b0f2-47ba2b356d99" containerName="extract" Jul 07 19:08:31.545000 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.545002 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a2cbdd2-093d-495d-af87-de9ea1ec7e45" containerName="util" Jul 07 19:08:31.545000 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.545009 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a2cbdd2-093d-495d-af87-de9ea1ec7e45" containerName="util" Jul 07 19:08:31.545772 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.545124 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a2cbdd2-093d-495d-af87-de9ea1ec7e45" containerName="extract" Jul 07 19:08:31.545772 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.545135 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c8b8b63-d0c6-437a-b0db-525553f775d0" containerName="extract" Jul 07 19:08:31.545772 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.545147 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b1b73ff-28a6-4e52-85e0-df7d72f95cb7" containerName="extract" Jul 07 19:08:31.545772 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.545156 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="384eb23f-f99e-4d93-b0f2-47ba2b356d99" containerName="extract" Jul 07 19:08:31.548399 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.548364 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wv44m" Jul 07 19:08:31.550571 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.550545 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 07 19:08:31.550750 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.550734 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 07 19:08:31.550813 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.550735 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 19:08:31.551268 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.551240 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 19:08:31.551352 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.551265 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-f2sxg\"" Jul 07 19:08:31.562144 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.562118 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-wv44m"] Jul 07 19:08:31.636256 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.636221 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb309a1e-80c0-4399-861e-2a689c8ad613-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-wv44m\" (UID: \"fb309a1e-80c0-4399-861e-2a689c8ad613\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wv44m" Jul 07 19:08:31.636404 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.636261 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb309a1e-80c0-4399-861e-2a689c8ad613-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-wv44m\" (UID: \"fb309a1e-80c0-4399-861e-2a689c8ad613\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wv44m" Jul 07 19:08:31.636404 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.636281 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4lj7\" (UniqueName: \"kubernetes.io/projected/fb309a1e-80c0-4399-861e-2a689c8ad613-kube-api-access-d4lj7\") pod \"kuadrant-console-plugin-66b5774985-wv44m\" (UID: \"fb309a1e-80c0-4399-861e-2a689c8ad613\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wv44m" Jul 07 19:08:31.737153 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.737115 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb309a1e-80c0-4399-861e-2a689c8ad613-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-wv44m\" (UID: \"fb309a1e-80c0-4399-861e-2a689c8ad613\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wv44m" Jul 07 19:08:31.737153 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.737153 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4lj7\" (UniqueName: \"kubernetes.io/projected/fb309a1e-80c0-4399-861e-2a689c8ad613-kube-api-access-d4lj7\") pod \"kuadrant-console-plugin-66b5774985-wv44m\" (UID: \"fb309a1e-80c0-4399-861e-2a689c8ad613\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wv44m" Jul 07 19:08:31.737370 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.737347 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb309a1e-80c0-4399-861e-2a689c8ad613-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-wv44m\" (UID: \"fb309a1e-80c0-4399-861e-2a689c8ad613\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wv44m" Jul 07 19:08:31.737906 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.737888 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/fb309a1e-80c0-4399-861e-2a689c8ad613-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-wv44m\" (UID: \"fb309a1e-80c0-4399-861e-2a689c8ad613\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wv44m" Jul 07 19:08:31.739558 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.739540 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb309a1e-80c0-4399-861e-2a689c8ad613-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-wv44m\" (UID: \"fb309a1e-80c0-4399-861e-2a689c8ad613\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wv44m" Jul 07 19:08:31.754281 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.754251 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4lj7\" (UniqueName: \"kubernetes.io/projected/fb309a1e-80c0-4399-861e-2a689c8ad613-kube-api-access-d4lj7\") pod \"kuadrant-console-plugin-66b5774985-wv44m\" (UID: \"fb309a1e-80c0-4399-861e-2a689c8ad613\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wv44m" Jul 07 19:08:31.858415 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:31.858330 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wv44m" Jul 07 19:08:32.008330 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:32.005733 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-wv44m"] Jul 07 19:08:32.569663 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:32.569624 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wv44m" event={"ID":"fb309a1e-80c0-4399-861e-2a689c8ad613","Type":"ContainerStarted","Data":"a3ded7a95c809864a689c25d0a58ab6a2ad3baad468a8bdee08557253b3d92f8"} Jul 07 19:08:45.337010 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.336964 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58d6f4c969-pm94w"] Jul 07 19:08:45.376147 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.376115 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58d6f4c969-pm94w"] Jul 07 19:08:45.376317 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.376241 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.378848 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.378819 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 19:08:45.378984 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.378851 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6wjfm\"" Jul 07 19:08:45.378984 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.378851 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 19:08:45.379199 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.379181 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 19:08:45.379781 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.379654 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 19:08:45.379781 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.379723 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 19:08:45.384566 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.384543 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 19:08:45.468582 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.468548 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc80c657-dc92-43e9-9f27-89c656321e52-console-serving-cert\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.468786 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.468589 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc80c657-dc92-43e9-9f27-89c656321e52-oauth-serving-cert\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.468786 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.468641 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc80c657-dc92-43e9-9f27-89c656321e52-console-config\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.468786 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.468668 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc80c657-dc92-43e9-9f27-89c656321e52-trusted-ca-bundle\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.468786 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.468756 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xst7\" (UniqueName: \"kubernetes.io/projected/bc80c657-dc92-43e9-9f27-89c656321e52-kube-api-access-5xst7\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.468995 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.468806 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc80c657-dc92-43e9-9f27-89c656321e52-service-ca\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.468995 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.468887 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc80c657-dc92-43e9-9f27-89c656321e52-console-oauth-config\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.569553 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.569515 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc80c657-dc92-43e9-9f27-89c656321e52-console-serving-cert\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.569553 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.569559 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc80c657-dc92-43e9-9f27-89c656321e52-oauth-serving-cert\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.569804 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.569577 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc80c657-dc92-43e9-9f27-89c656321e52-console-config\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.569804 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.569601 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc80c657-dc92-43e9-9f27-89c656321e52-trusted-ca-bundle\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.569804 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.569642 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xst7\" (UniqueName: \"kubernetes.io/projected/bc80c657-dc92-43e9-9f27-89c656321e52-kube-api-access-5xst7\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.569804 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.569669 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc80c657-dc92-43e9-9f27-89c656321e52-service-ca\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.569804 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.569736 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc80c657-dc92-43e9-9f27-89c656321e52-console-oauth-config\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.570469 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.570441 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc80c657-dc92-43e9-9f27-89c656321e52-console-config\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.570606 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.570480 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc80c657-dc92-43e9-9f27-89c656321e52-oauth-serving-cert\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.570606 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.570578 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc80c657-dc92-43e9-9f27-89c656321e52-trusted-ca-bundle\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.570738 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.570682 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc80c657-dc92-43e9-9f27-89c656321e52-service-ca\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.572580 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.572557 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc80c657-dc92-43e9-9f27-89c656321e52-console-serving-cert\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.572669 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.572583 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc80c657-dc92-43e9-9f27-89c656321e52-console-oauth-config\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.583534 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.583499 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xst7\" (UniqueName: \"kubernetes.io/projected/bc80c657-dc92-43e9-9f27-89c656321e52-kube-api-access-5xst7\") pod \"console-58d6f4c969-pm94w\" (UID: \"bc80c657-dc92-43e9-9f27-89c656321e52\") " pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:45.689332 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:45.689162 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:08:56.623147 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:56.623121 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58d6f4c969-pm94w"] Jul 07 19:08:56.624899 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:08:56.624870 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc80c657_dc92_43e9_9f27_89c656321e52.slice/crio-47b30f1c39c6360d7446a96d8ee0c4d9c2800ac7d3dee083a4ecbb3a22fc9446 WatchSource:0}: Error finding container 47b30f1c39c6360d7446a96d8ee0c4d9c2800ac7d3dee083a4ecbb3a22fc9446: Status 404 returned error can't find the container with id 47b30f1c39c6360d7446a96d8ee0c4d9c2800ac7d3dee083a4ecbb3a22fc9446 Jul 07 19:08:56.710765 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:56.710730 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wv44m" event={"ID":"fb309a1e-80c0-4399-861e-2a689c8ad613","Type":"ContainerStarted","Data":"14e4ab37b39454bb846e469c3db5f9be538fb03fc91efabcf304c75c7e62b56f"} Jul 07 19:08:56.712433 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:56.712406 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d6f4c969-pm94w" event={"ID":"bc80c657-dc92-43e9-9f27-89c656321e52","Type":"ContainerStarted","Data":"674b7bf8df004ef2ca503ce6f38fdc14bd1f5b44e927a69dce3f5c47e5dafa4f"} Jul 07 19:08:56.712557 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:56.712454 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d6f4c969-pm94w" event={"ID":"bc80c657-dc92-43e9-9f27-89c656321e52","Type":"ContainerStarted","Data":"47b30f1c39c6360d7446a96d8ee0c4d9c2800ac7d3dee083a4ecbb3a22fc9446"} Jul 07 19:08:56.728151 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:56.728103 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-wv44m" podStartSLOduration=1.182175886 podStartE2EDuration="25.728086383s" totalImagesPullingTime="24.545910497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:56.727907741 +0000 UTC m=+506.920447810" watchObservedRunningTime="2026-07-07 19:08:56.728086383 +0000 UTC m=+506.920626452" Jul 07 19:08:56.751572 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:08:56.751475 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58d6f4c969-pm94w" podStartSLOduration=11.751458816 podStartE2EDuration="11.751458816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:56.748939263 +0000 UTC m=+506.941479351" watchObservedRunningTime="2026-07-07 19:08:56.751458816 +0000 UTC m=+506.943998883" Jul 07 19:09:05.690407 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:05.690360 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:09:05.690922 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:05.690454 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:09:05.695191 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:05.695165 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:09:05.767241 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:05.767210 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58d6f4c969-pm94w" Jul 07 19:09:14.483219 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:14.483180 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:09:14.603682 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:14.603650 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:09:14.603840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:14.603697 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:09:14.603840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:14.603761 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-vmrzr" Jul 07 19:09:14.606085 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:14.606040 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 07 19:09:14.725376 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:14.725333 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cbe5dc1c-b246-496e-b9ad-52598990970b-config-file\") pod \"limitador-limitador-54d6d988d5-vmrzr\" (UID: \"cbe5dc1c-b246-496e-b9ad-52598990970b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vmrzr" Jul 07 19:09:14.725554 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:14.725393 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzshw\" (UniqueName: \"kubernetes.io/projected/cbe5dc1c-b246-496e-b9ad-52598990970b-kube-api-access-xzshw\") pod \"limitador-limitador-54d6d988d5-vmrzr\" (UID: \"cbe5dc1c-b246-496e-b9ad-52598990970b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vmrzr" Jul 07 19:09:14.825907 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:14.825826 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cbe5dc1c-b246-496e-b9ad-52598990970b-config-file\") pod \"limitador-limitador-54d6d988d5-vmrzr\" (UID: \"cbe5dc1c-b246-496e-b9ad-52598990970b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vmrzr" Jul 07 19:09:14.825907 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:14.825879 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzshw\" (UniqueName: \"kubernetes.io/projected/cbe5dc1c-b246-496e-b9ad-52598990970b-kube-api-access-xzshw\") pod \"limitador-limitador-54d6d988d5-vmrzr\" (UID: \"cbe5dc1c-b246-496e-b9ad-52598990970b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vmrzr" Jul 07 19:09:14.826546 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:14.826522 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cbe5dc1c-b246-496e-b9ad-52598990970b-config-file\") pod \"limitador-limitador-54d6d988d5-vmrzr\" (UID: \"cbe5dc1c-b246-496e-b9ad-52598990970b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vmrzr" Jul 07 19:09:14.834456 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:14.834431 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzshw\" (UniqueName: \"kubernetes.io/projected/cbe5dc1c-b246-496e-b9ad-52598990970b-kube-api-access-xzshw\") pod \"limitador-limitador-54d6d988d5-vmrzr\" (UID: \"cbe5dc1c-b246-496e-b9ad-52598990970b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-vmrzr" Jul 07 19:09:14.915431 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:14.915395 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-vmrzr" Jul 07 19:09:14.977997 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:14.977963 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-2m5br"] Jul 07 19:09:15.007184 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.007145 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-2m5br"] Jul 07 19:09:15.007326 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.007267 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-2m5br" Jul 07 19:09:15.009909 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.009887 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kj54w\"" Jul 07 19:09:15.091905 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.091874 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:09:15.093255 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:09:15.093214 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe5dc1c_b246_496e_b9ad_52598990970b.slice/crio-3402f9b52bda4ac6e2a63c56db7d7dc1f793fa7f7570471b203189919d4eefa4 WatchSource:0}: Error finding container 3402f9b52bda4ac6e2a63c56db7d7dc1f793fa7f7570471b203189919d4eefa4: Status 404 returned error can't find the container with id 3402f9b52bda4ac6e2a63c56db7d7dc1f793fa7f7570471b203189919d4eefa4 Jul 07 19:09:15.128348 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.128319 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2bpx\" (UniqueName: \"kubernetes.io/projected/57e0c92b-4404-42fb-947f-0a7265a61c44-kube-api-access-p2bpx\") pod \"authorino-58cd955675-2m5br\" (UID: \"57e0c92b-4404-42fb-947f-0a7265a61c44\") " pod="kuadrant-system/authorino-58cd955675-2m5br" Jul 07 19:09:15.225928 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.225891 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-zmskz"] Jul 07 19:09:15.229192 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.229157 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2bpx\" (UniqueName: \"kubernetes.io/projected/57e0c92b-4404-42fb-947f-0a7265a61c44-kube-api-access-p2bpx\") pod \"authorino-58cd955675-2m5br\" (UID: \"57e0c92b-4404-42fb-947f-0a7265a61c44\") " pod="kuadrant-system/authorino-58cd955675-2m5br" Jul 07 19:09:15.241683 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.241646 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2bpx\" (UniqueName: \"kubernetes.io/projected/57e0c92b-4404-42fb-947f-0a7265a61c44-kube-api-access-p2bpx\") pod \"authorino-58cd955675-2m5br\" (UID: \"57e0c92b-4404-42fb-947f-0a7265a61c44\") " pod="kuadrant-system/authorino-58cd955675-2m5br" Jul 07 19:09:15.254659 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.254630 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zmskz"] Jul 07 19:09:15.254774 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.254725 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zmskz" Jul 07 19:09:15.321569 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.321536 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-2m5br" Jul 07 19:09:15.330057 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.330009 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf7x9\" (UniqueName: \"kubernetes.io/projected/9f330977-93a1-4619-a882-78d6bfbcb8bf-kube-api-access-wf7x9\") pod \"authorino-75d556bd77-zmskz\" (UID: \"9f330977-93a1-4619-a882-78d6bfbcb8bf\") " pod="kuadrant-system/authorino-75d556bd77-zmskz" Jul 07 19:09:15.430708 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.430670 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf7x9\" (UniqueName: \"kubernetes.io/projected/9f330977-93a1-4619-a882-78d6bfbcb8bf-kube-api-access-wf7x9\") pod \"authorino-75d556bd77-zmskz\" (UID: \"9f330977-93a1-4619-a882-78d6bfbcb8bf\") " pod="kuadrant-system/authorino-75d556bd77-zmskz" Jul 07 19:09:15.440334 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.440309 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf7x9\" (UniqueName: \"kubernetes.io/projected/9f330977-93a1-4619-a882-78d6bfbcb8bf-kube-api-access-wf7x9\") pod \"authorino-75d556bd77-zmskz\" (UID: \"9f330977-93a1-4619-a882-78d6bfbcb8bf\") " pod="kuadrant-system/authorino-75d556bd77-zmskz" Jul 07 19:09:15.475526 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.475501 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-2m5br"] Jul 07 19:09:15.477247 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:09:15.477218 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57e0c92b_4404_42fb_947f_0a7265a61c44.slice/crio-c42616ae0ed367a183d5170e4a4f406c3128646c7ce29dd811f9a368e49b4d29 WatchSource:0}: Error finding container c42616ae0ed367a183d5170e4a4f406c3128646c7ce29dd811f9a368e49b4d29: Status 404 returned error can't find the container with id c42616ae0ed367a183d5170e4a4f406c3128646c7ce29dd811f9a368e49b4d29 Jul 07 19:09:15.563633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.563590 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zmskz" Jul 07 19:09:15.756144 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.756076 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zmskz"] Jul 07 19:09:15.757470 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:09:15.757431 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f330977_93a1_4619_a882_78d6bfbcb8bf.slice/crio-703ad7ecec4d6f2bc899626018de9ded83c16b319be61ffa83f0025a916d3b63 WatchSource:0}: Error finding container 703ad7ecec4d6f2bc899626018de9ded83c16b319be61ffa83f0025a916d3b63: Status 404 returned error can't find the container with id 703ad7ecec4d6f2bc899626018de9ded83c16b319be61ffa83f0025a916d3b63 Jul 07 19:09:15.816761 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.816725 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zmskz" event={"ID":"9f330977-93a1-4619-a882-78d6bfbcb8bf","Type":"ContainerStarted","Data":"703ad7ecec4d6f2bc899626018de9ded83c16b319be61ffa83f0025a916d3b63"} Jul 07 19:09:15.818396 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.818364 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-2m5br" event={"ID":"57e0c92b-4404-42fb-947f-0a7265a61c44","Type":"ContainerStarted","Data":"c42616ae0ed367a183d5170e4a4f406c3128646c7ce29dd811f9a368e49b4d29"} Jul 07 19:09:15.819813 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:15.819780 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-vmrzr" event={"ID":"cbe5dc1c-b246-496e-b9ad-52598990970b","Type":"ContainerStarted","Data":"3402f9b52bda4ac6e2a63c56db7d7dc1f793fa7f7570471b203189919d4eefa4"} Jul 07 19:09:21.859901 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:21.859785 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zmskz" event={"ID":"9f330977-93a1-4619-a882-78d6bfbcb8bf","Type":"ContainerStarted","Data":"16cf79c11e06a2c867bde293280dc36f4306f5428cfde48a4a130499dab7216e"} Jul 07 19:09:21.861632 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:21.861591 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-2m5br" event={"ID":"57e0c92b-4404-42fb-947f-0a7265a61c44","Type":"ContainerStarted","Data":"5c162c58d696226be30b3cf0b3bb310606cbeede9eced1f9a6f4910f3bda45a9"} Jul 07 19:09:21.863384 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:21.863358 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-vmrzr" event={"ID":"cbe5dc1c-b246-496e-b9ad-52598990970b","Type":"ContainerStarted","Data":"0a64b2c63238ba5183dca13de3ad3185cb9e59a74099a4ec4424099903e08c4c"} Jul 07 19:09:21.863503 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:21.863491 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-vmrzr" Jul 07 19:09:21.878689 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:21.878636 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-zmskz" podStartSLOduration=1.116854934 podStartE2EDuration="6.878619987s" totalImagesPullingTime="5.761765053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:21.878289467 +0000 UTC m=+532.070829535" watchObservedRunningTime="2026-07-07 19:09:21.878619987 +0000 UTC m=+532.071160032" Jul 07 19:09:21.895801 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:21.895738 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-vmrzr" podStartSLOduration=1.469303306 podStartE2EDuration="7.895717443s" totalImagesPullingTime="6.426414137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:21.895414434 +0000 UTC m=+532.087954500" watchObservedRunningTime="2026-07-07 19:09:21.895717443 +0000 UTC m=+532.088257511" Jul 07 19:09:21.921091 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:21.921016 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-2m5br" podStartSLOduration=1.863657149 podStartE2EDuration="7.920996725s" totalImagesPullingTime="6.057339576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:21.918119102 +0000 UTC m=+532.110659169" watchObservedRunningTime="2026-07-07 19:09:21.920996725 +0000 UTC m=+532.113536792" Jul 07 19:09:21.935713 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:21.935677 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-2m5br"] Jul 07 19:09:23.874847 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:23.874762 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-2m5br" podUID="57e0c92b-4404-42fb-947f-0a7265a61c44" containerName="authorino" containerID="cri-o://5c162c58d696226be30b3cf0b3bb310606cbeede9eced1f9a6f4910f3bda45a9" gracePeriod=30 Jul 07 19:09:24.132994 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:24.132972 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-2m5br" Jul 07 19:09:24.218053 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:24.218017 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2bpx\" (UniqueName: \"kubernetes.io/projected/57e0c92b-4404-42fb-947f-0a7265a61c44-kube-api-access-p2bpx\") pod \"57e0c92b-4404-42fb-947f-0a7265a61c44\" (UID: \"57e0c92b-4404-42fb-947f-0a7265a61c44\") " Jul 07 19:09:24.220120 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:24.220095 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e0c92b-4404-42fb-947f-0a7265a61c44-kube-api-access-p2bpx" (OuterVolumeSpecName: "kube-api-access-p2bpx") pod "57e0c92b-4404-42fb-947f-0a7265a61c44" (UID: "57e0c92b-4404-42fb-947f-0a7265a61c44"). InnerVolumeSpecName "kube-api-access-p2bpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:09:24.318675 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:24.318642 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2bpx\" (UniqueName: \"kubernetes.io/projected/57e0c92b-4404-42fb-947f-0a7265a61c44-kube-api-access-p2bpx\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:09:24.880405 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:24.880370 2568 generic.go:356] "Generic (PLEG): container finished" podID="57e0c92b-4404-42fb-947f-0a7265a61c44" containerID="5c162c58d696226be30b3cf0b3bb310606cbeede9eced1f9a6f4910f3bda45a9" exitCode=0 Jul 07 19:09:24.880811 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:24.880418 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-2m5br" Jul 07 19:09:24.880811 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:24.880461 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-2m5br" event={"ID":"57e0c92b-4404-42fb-947f-0a7265a61c44","Type":"ContainerDied","Data":"5c162c58d696226be30b3cf0b3bb310606cbeede9eced1f9a6f4910f3bda45a9"} Jul 07 19:09:24.880811 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:24.880508 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-2m5br" event={"ID":"57e0c92b-4404-42fb-947f-0a7265a61c44","Type":"ContainerDied","Data":"c42616ae0ed367a183d5170e4a4f406c3128646c7ce29dd811f9a368e49b4d29"} Jul 07 19:09:24.880811 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:24.880529 2568 scope.go:117] "RemoveContainer" containerID="5c162c58d696226be30b3cf0b3bb310606cbeede9eced1f9a6f4910f3bda45a9" Jul 07 19:09:24.889232 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:24.889217 2568 scope.go:117] "RemoveContainer" containerID="5c162c58d696226be30b3cf0b3bb310606cbeede9eced1f9a6f4910f3bda45a9" Jul 07 19:09:24.889511 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:09:24.889492 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c162c58d696226be30b3cf0b3bb310606cbeede9eced1f9a6f4910f3bda45a9\": container with ID starting with 5c162c58d696226be30b3cf0b3bb310606cbeede9eced1f9a6f4910f3bda45a9 not found: ID does not exist" containerID="5c162c58d696226be30b3cf0b3bb310606cbeede9eced1f9a6f4910f3bda45a9" Jul 07 19:09:24.889555 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:24.889520 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c162c58d696226be30b3cf0b3bb310606cbeede9eced1f9a6f4910f3bda45a9"} err="failed to get container status \"5c162c58d696226be30b3cf0b3bb310606cbeede9eced1f9a6f4910f3bda45a9\": rpc error: code = NotFound desc = could not find container \"5c162c58d696226be30b3cf0b3bb310606cbeede9eced1f9a6f4910f3bda45a9\": container with ID starting with 5c162c58d696226be30b3cf0b3bb310606cbeede9eced1f9a6f4910f3bda45a9 not found: ID does not exist" Jul 07 19:09:24.907947 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:24.907909 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-2m5br"] Jul 07 19:09:24.913018 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:24.912995 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-2m5br"] Jul 07 19:09:26.329303 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:26.329256 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57e0c92b-4404-42fb-947f-0a7265a61c44" path="/var/lib/kubelet/pods/57e0c92b-4404-42fb-947f-0a7265a61c44/volumes" Jul 07 19:09:32.872208 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:32.872177 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-vmrzr" Jul 07 19:09:47.229654 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.229619 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-h4jhr"] Jul 07 19:09:47.230040 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.229971 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e0c92b-4404-42fb-947f-0a7265a61c44" containerName="authorino" Jul 07 19:09:47.230040 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.229982 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e0c92b-4404-42fb-947f-0a7265a61c44" containerName="authorino" Jul 07 19:09:47.230131 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.230065 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e0c92b-4404-42fb-947f-0a7265a61c44" containerName="authorino" Jul 07 19:09:47.260722 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.260690 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-h4jhr" Jul 07 19:09:47.267949 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.267612 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-h4jhr"] Jul 07 19:09:47.318957 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.318920 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zx9s\" (UniqueName: \"kubernetes.io/projected/488f97d9-bb43-4465-9f05-e34efab6a2f0-kube-api-access-9zx9s\") pod \"authorino-5d4b67c446-h4jhr\" (UID: \"488f97d9-bb43-4465-9f05-e34efab6a2f0\") " pod="kuadrant-system/authorino-5d4b67c446-h4jhr" Jul 07 19:09:47.419896 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.419852 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zx9s\" (UniqueName: \"kubernetes.io/projected/488f97d9-bb43-4465-9f05-e34efab6a2f0-kube-api-access-9zx9s\") pod \"authorino-5d4b67c446-h4jhr\" (UID: \"488f97d9-bb43-4465-9f05-e34efab6a2f0\") " pod="kuadrant-system/authorino-5d4b67c446-h4jhr" Jul 07 19:09:47.431319 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.431283 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zx9s\" (UniqueName: \"kubernetes.io/projected/488f97d9-bb43-4465-9f05-e34efab6a2f0-kube-api-access-9zx9s\") pod \"authorino-5d4b67c446-h4jhr\" (UID: \"488f97d9-bb43-4465-9f05-e34efab6a2f0\") " pod="kuadrant-system/authorino-5d4b67c446-h4jhr" Jul 07 19:09:47.467147 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.467114 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-h4jhr"] Jul 07 19:09:47.467387 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.467375 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-h4jhr" Jul 07 19:09:47.501432 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.501351 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d7bbb74fc-zp9gh"] Jul 07 19:09:47.505029 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.505004 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d7bbb74fc-zp9gh" Jul 07 19:09:47.516497 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.516468 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d7bbb74fc-zp9gh"] Jul 07 19:09:47.610839 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.610814 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-h4jhr"] Jul 07 19:09:47.613617 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:09:47.613588 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod488f97d9_bb43_4465_9f05_e34efab6a2f0.slice/crio-c238c96bdad759549bfb261d70e4bcee91d6d8ffd242b16d5410789c9473dfef WatchSource:0}: Error finding container c238c96bdad759549bfb261d70e4bcee91d6d8ffd242b16d5410789c9473dfef: Status 404 returned error can't find the container with id c238c96bdad759549bfb261d70e4bcee91d6d8ffd242b16d5410789c9473dfef Jul 07 19:09:47.622437 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.622407 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpz6p\" (UniqueName: \"kubernetes.io/projected/73dc6e9f-dfb3-4504-9e21-a733a898911f-kube-api-access-rpz6p\") pod \"authorino-d7bbb74fc-zp9gh\" (UID: \"73dc6e9f-dfb3-4504-9e21-a733a898911f\") " pod="kuadrant-system/authorino-d7bbb74fc-zp9gh" Jul 07 19:09:47.724014 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.723980 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpz6p\" (UniqueName: \"kubernetes.io/projected/73dc6e9f-dfb3-4504-9e21-a733a898911f-kube-api-access-rpz6p\") pod \"authorino-d7bbb74fc-zp9gh\" (UID: \"73dc6e9f-dfb3-4504-9e21-a733a898911f\") " pod="kuadrant-system/authorino-d7bbb74fc-zp9gh" Jul 07 19:09:47.734708 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.734671 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d7bbb74fc-zp9gh"] Jul 07 19:09:47.734974 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:09:47.734955 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rpz6p], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-d7bbb74fc-zp9gh" podUID="73dc6e9f-dfb3-4504-9e21-a733a898911f" Jul 07 19:09:47.736015 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.735993 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpz6p\" (UniqueName: \"kubernetes.io/projected/73dc6e9f-dfb3-4504-9e21-a733a898911f-kube-api-access-rpz6p\") pod \"authorino-d7bbb74fc-zp9gh\" (UID: \"73dc6e9f-dfb3-4504-9e21-a733a898911f\") " pod="kuadrant-system/authorino-d7bbb74fc-zp9gh" Jul 07 19:09:47.768266 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.768186 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75c97d878f-mp27s"] Jul 07 19:09:47.771637 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.771618 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75c97d878f-mp27s" Jul 07 19:09:47.773829 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.773799 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 07 19:09:47.783661 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.783636 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75c97d878f-mp27s"] Jul 07 19:09:47.825409 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.825372 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dztpd\" (UniqueName: \"kubernetes.io/projected/5c8f9f03-4966-4237-91ef-245fbaadc7f4-kube-api-access-dztpd\") pod \"authorino-75c97d878f-mp27s\" (UID: \"5c8f9f03-4966-4237-91ef-245fbaadc7f4\") " pod="kuadrant-system/authorino-75c97d878f-mp27s" Jul 07 19:09:47.825575 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.825491 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5c8f9f03-4966-4237-91ef-245fbaadc7f4-tls-cert\") pod \"authorino-75c97d878f-mp27s\" (UID: \"5c8f9f03-4966-4237-91ef-245fbaadc7f4\") " pod="kuadrant-system/authorino-75c97d878f-mp27s" Jul 07 19:09:47.926309 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.926269 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5c8f9f03-4966-4237-91ef-245fbaadc7f4-tls-cert\") pod \"authorino-75c97d878f-mp27s\" (UID: \"5c8f9f03-4966-4237-91ef-245fbaadc7f4\") " pod="kuadrant-system/authorino-75c97d878f-mp27s" Jul 07 19:09:47.926459 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.926320 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dztpd\" (UniqueName: \"kubernetes.io/projected/5c8f9f03-4966-4237-91ef-245fbaadc7f4-kube-api-access-dztpd\") pod \"authorino-75c97d878f-mp27s\" (UID: \"5c8f9f03-4966-4237-91ef-245fbaadc7f4\") " pod="kuadrant-system/authorino-75c97d878f-mp27s" Jul 07 19:09:47.928714 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.928693 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5c8f9f03-4966-4237-91ef-245fbaadc7f4-tls-cert\") pod \"authorino-75c97d878f-mp27s\" (UID: \"5c8f9f03-4966-4237-91ef-245fbaadc7f4\") " pod="kuadrant-system/authorino-75c97d878f-mp27s" Jul 07 19:09:47.936821 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.936801 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dztpd\" (UniqueName: \"kubernetes.io/projected/5c8f9f03-4966-4237-91ef-245fbaadc7f4-kube-api-access-dztpd\") pod \"authorino-75c97d878f-mp27s\" (UID: \"5c8f9f03-4966-4237-91ef-245fbaadc7f4\") " pod="kuadrant-system/authorino-75c97d878f-mp27s" Jul 07 19:09:47.999131 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.999099 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d7bbb74fc-zp9gh" Jul 07 19:09:47.999296 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:47.999090 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-h4jhr" event={"ID":"488f97d9-bb43-4465-9f05-e34efab6a2f0","Type":"ContainerStarted","Data":"c238c96bdad759549bfb261d70e4bcee91d6d8ffd242b16d5410789c9473dfef"} Jul 07 19:09:48.005568 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:48.005544 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d7bbb74fc-zp9gh" Jul 07 19:09:48.026840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:48.026751 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpz6p\" (UniqueName: \"kubernetes.io/projected/73dc6e9f-dfb3-4504-9e21-a733a898911f-kube-api-access-rpz6p\") pod \"73dc6e9f-dfb3-4504-9e21-a733a898911f\" (UID: \"73dc6e9f-dfb3-4504-9e21-a733a898911f\") " Jul 07 19:09:48.028795 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:48.028745 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73dc6e9f-dfb3-4504-9e21-a733a898911f-kube-api-access-rpz6p" (OuterVolumeSpecName: "kube-api-access-rpz6p") pod "73dc6e9f-dfb3-4504-9e21-a733a898911f" (UID: "73dc6e9f-dfb3-4504-9e21-a733a898911f"). InnerVolumeSpecName "kube-api-access-rpz6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:09:48.082296 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:48.082251 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75c97d878f-mp27s" Jul 07 19:09:48.127817 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:48.127782 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpz6p\" (UniqueName: \"kubernetes.io/projected/73dc6e9f-dfb3-4504-9e21-a733a898911f-kube-api-access-rpz6p\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:09:48.234112 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:48.234080 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75c97d878f-mp27s"] Jul 07 19:09:48.235554 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:09:48.235525 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c8f9f03_4966_4237_91ef_245fbaadc7f4.slice/crio-e0dbccae3938c880631ce70c4b91ac0c628321320848515c62d3a4f80009ad22 WatchSource:0}: Error finding container e0dbccae3938c880631ce70c4b91ac0c628321320848515c62d3a4f80009ad22: Status 404 returned error can't find the container with id e0dbccae3938c880631ce70c4b91ac0c628321320848515c62d3a4f80009ad22 Jul 07 19:09:49.006650 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:49.006613 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-h4jhr" podUID="488f97d9-bb43-4465-9f05-e34efab6a2f0" containerName="authorino" containerID="cri-o://1eabe79c7811127ecfd480cc247f871edfbb7ab959f183f2382778f654eb519d" gracePeriod=30 Jul 07 19:09:49.006846 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:49.006620 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-h4jhr" event={"ID":"488f97d9-bb43-4465-9f05-e34efab6a2f0","Type":"ContainerStarted","Data":"1eabe79c7811127ecfd480cc247f871edfbb7ab959f183f2382778f654eb519d"} Jul 07 19:09:49.008320 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:49.008291 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75c97d878f-mp27s" event={"ID":"5c8f9f03-4966-4237-91ef-245fbaadc7f4","Type":"ContainerStarted","Data":"9dec83533c0c56adf7b86f9ba114e9fbd15991b36dc06eca0eca4dc73ecd573f"} Jul 07 19:09:49.008450 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:49.008326 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75c97d878f-mp27s" event={"ID":"5c8f9f03-4966-4237-91ef-245fbaadc7f4","Type":"ContainerStarted","Data":"e0dbccae3938c880631ce70c4b91ac0c628321320848515c62d3a4f80009ad22"} Jul 07 19:09:49.008450 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:49.008357 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d7bbb74fc-zp9gh" Jul 07 19:09:49.028463 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:49.028414 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-h4jhr" podStartSLOduration=1.533032999 podStartE2EDuration="2.028400509s" totalImagesPullingTime="495.36751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:49.026829123 +0000 UTC m=+559.219369189" watchObservedRunningTime="2026-07-07 19:09:49.028400509 +0000 UTC m=+559.220940576" Jul 07 19:09:49.058103 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:49.058069 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d7bbb74fc-zp9gh"] Jul 07 19:09:49.061864 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:49.061837 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d7bbb74fc-zp9gh"] Jul 07 19:09:49.079956 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:49.079902 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75c97d878f-mp27s" podStartSLOduration=1.753634039 podStartE2EDuration="2.079880917s" totalImagesPullingTime="326.246878ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:49.079776596 +0000 UTC m=+559.272316674" watchObservedRunningTime="2026-07-07 19:09:49.079880917 +0000 UTC m=+559.272420984" Jul 07 19:09:49.125533 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:49.125498 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zmskz"] Jul 07 19:09:49.125744 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:49.125718 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-zmskz" podUID="9f330977-93a1-4619-a882-78d6bfbcb8bf" containerName="authorino" containerID="cri-o://16cf79c11e06a2c867bde293280dc36f4306f5428cfde48a4a130499dab7216e" gracePeriod=30 Jul 07 19:09:49.247614 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:49.247591 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-h4jhr" Jul 07 19:09:49.337236 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:49.337207 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zx9s\" (UniqueName: \"kubernetes.io/projected/488f97d9-bb43-4465-9f05-e34efab6a2f0-kube-api-access-9zx9s\") pod \"488f97d9-bb43-4465-9f05-e34efab6a2f0\" (UID: \"488f97d9-bb43-4465-9f05-e34efab6a2f0\") " Jul 07 19:09:49.340651 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:49.340625 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488f97d9-bb43-4465-9f05-e34efab6a2f0-kube-api-access-9zx9s" (OuterVolumeSpecName: "kube-api-access-9zx9s") pod "488f97d9-bb43-4465-9f05-e34efab6a2f0" (UID: "488f97d9-bb43-4465-9f05-e34efab6a2f0"). InnerVolumeSpecName "kube-api-access-9zx9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:09:49.368470 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:49.368446 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zmskz" Jul 07 19:09:49.437903 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:49.437869 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf7x9\" (UniqueName: \"kubernetes.io/projected/9f330977-93a1-4619-a882-78d6bfbcb8bf-kube-api-access-wf7x9\") pod \"9f330977-93a1-4619-a882-78d6bfbcb8bf\" (UID: \"9f330977-93a1-4619-a882-78d6bfbcb8bf\") " Jul 07 19:09:49.438651 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:49.438605 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zx9s\" (UniqueName: \"kubernetes.io/projected/488f97d9-bb43-4465-9f05-e34efab6a2f0-kube-api-access-9zx9s\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:09:49.440275 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:49.440253 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f330977-93a1-4619-a882-78d6bfbcb8bf-kube-api-access-wf7x9" (OuterVolumeSpecName: "kube-api-access-wf7x9") pod "9f330977-93a1-4619-a882-78d6bfbcb8bf" (UID: "9f330977-93a1-4619-a882-78d6bfbcb8bf"). InnerVolumeSpecName "kube-api-access-wf7x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:09:49.539214 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:49.539183 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf7x9\" (UniqueName: \"kubernetes.io/projected/9f330977-93a1-4619-a882-78d6bfbcb8bf-kube-api-access-wf7x9\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:09:50.014813 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.014776 2568 generic.go:356] "Generic (PLEG): container finished" podID="9f330977-93a1-4619-a882-78d6bfbcb8bf" containerID="16cf79c11e06a2c867bde293280dc36f4306f5428cfde48a4a130499dab7216e" exitCode=0 Jul 07 19:09:50.014976 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.014820 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zmskz" Jul 07 19:09:50.014976 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.014838 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zmskz" event={"ID":"9f330977-93a1-4619-a882-78d6bfbcb8bf","Type":"ContainerDied","Data":"16cf79c11e06a2c867bde293280dc36f4306f5428cfde48a4a130499dab7216e"} Jul 07 19:09:50.014976 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.014876 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zmskz" event={"ID":"9f330977-93a1-4619-a882-78d6bfbcb8bf","Type":"ContainerDied","Data":"703ad7ecec4d6f2bc899626018de9ded83c16b319be61ffa83f0025a916d3b63"} Jul 07 19:09:50.014976 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.014894 2568 scope.go:117] "RemoveContainer" containerID="16cf79c11e06a2c867bde293280dc36f4306f5428cfde48a4a130499dab7216e" Jul 07 19:09:50.016220 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.016205 2568 generic.go:356] "Generic (PLEG): container finished" podID="488f97d9-bb43-4465-9f05-e34efab6a2f0" containerID="1eabe79c7811127ecfd480cc247f871edfbb7ab959f183f2382778f654eb519d" exitCode=0 Jul 07 19:09:50.016301 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.016233 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-h4jhr" event={"ID":"488f97d9-bb43-4465-9f05-e34efab6a2f0","Type":"ContainerDied","Data":"1eabe79c7811127ecfd480cc247f871edfbb7ab959f183f2382778f654eb519d"} Jul 07 19:09:50.016301 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.016255 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-h4jhr" Jul 07 19:09:50.016301 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.016265 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-h4jhr" event={"ID":"488f97d9-bb43-4465-9f05-e34efab6a2f0","Type":"ContainerDied","Data":"c238c96bdad759549bfb261d70e4bcee91d6d8ffd242b16d5410789c9473dfef"} Jul 07 19:09:50.026468 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.026450 2568 scope.go:117] "RemoveContainer" containerID="16cf79c11e06a2c867bde293280dc36f4306f5428cfde48a4a130499dab7216e" Jul 07 19:09:50.026785 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:09:50.026753 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16cf79c11e06a2c867bde293280dc36f4306f5428cfde48a4a130499dab7216e\": container with ID starting with 16cf79c11e06a2c867bde293280dc36f4306f5428cfde48a4a130499dab7216e not found: ID does not exist" containerID="16cf79c11e06a2c867bde293280dc36f4306f5428cfde48a4a130499dab7216e" Jul 07 19:09:50.026882 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.026795 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16cf79c11e06a2c867bde293280dc36f4306f5428cfde48a4a130499dab7216e"} err="failed to get container status \"16cf79c11e06a2c867bde293280dc36f4306f5428cfde48a4a130499dab7216e\": rpc error: code = NotFound desc = could not find container \"16cf79c11e06a2c867bde293280dc36f4306f5428cfde48a4a130499dab7216e\": container with ID starting with 16cf79c11e06a2c867bde293280dc36f4306f5428cfde48a4a130499dab7216e not found: ID does not exist" Jul 07 19:09:50.026882 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.026820 2568 scope.go:117] "RemoveContainer" containerID="1eabe79c7811127ecfd480cc247f871edfbb7ab959f183f2382778f654eb519d" Jul 07 19:09:50.035209 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.035192 2568 scope.go:117] "RemoveContainer" containerID="1eabe79c7811127ecfd480cc247f871edfbb7ab959f183f2382778f654eb519d" Jul 07 19:09:50.035435 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:09:50.035417 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eabe79c7811127ecfd480cc247f871edfbb7ab959f183f2382778f654eb519d\": container with ID starting with 1eabe79c7811127ecfd480cc247f871edfbb7ab959f183f2382778f654eb519d not found: ID does not exist" containerID="1eabe79c7811127ecfd480cc247f871edfbb7ab959f183f2382778f654eb519d" Jul 07 19:09:50.035481 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.035443 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eabe79c7811127ecfd480cc247f871edfbb7ab959f183f2382778f654eb519d"} err="failed to get container status \"1eabe79c7811127ecfd480cc247f871edfbb7ab959f183f2382778f654eb519d\": rpc error: code = NotFound desc = could not find container \"1eabe79c7811127ecfd480cc247f871edfbb7ab959f183f2382778f654eb519d\": container with ID starting with 1eabe79c7811127ecfd480cc247f871edfbb7ab959f183f2382778f654eb519d not found: ID does not exist" Jul 07 19:09:50.054126 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.054102 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-h4jhr"] Jul 07 19:09:50.057965 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.057946 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-h4jhr"] Jul 07 19:09:50.075523 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.075494 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zmskz"] Jul 07 19:09:50.083240 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.083214 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zmskz"] Jul 07 19:09:50.329323 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.329240 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="488f97d9-bb43-4465-9f05-e34efab6a2f0" path="/var/lib/kubelet/pods/488f97d9-bb43-4465-9f05-e34efab6a2f0/volumes" Jul 07 19:09:50.329761 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.329699 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73dc6e9f-dfb3-4504-9e21-a733a898911f" path="/var/lib/kubelet/pods/73dc6e9f-dfb3-4504-9e21-a733a898911f/volumes" Jul 07 19:09:50.329967 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:09:50.329947 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f330977-93a1-4619-a882-78d6bfbcb8bf" path="/var/lib/kubelet/pods/9f330977-93a1-4619-a882-78d6bfbcb8bf/volumes" Jul 07 19:10:31.441902 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:10:31.441798 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:10:31.441902 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:10:31.441863 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:10:32.861793 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:10:32.861753 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:11:02.868613 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:11:02.868532 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:11:15.395114 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:11:15.395075 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:11:27.770837 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:11:27.770796 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:11:32.572972 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:11:32.572936 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:11:45.068230 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:11:45.068189 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:11:54.556568 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:11:54.556524 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:12:50.449193 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:50.449103 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c6b94b488-qmmng"] Jul 07 19:12:50.449783 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:50.449635 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="488f97d9-bb43-4465-9f05-e34efab6a2f0" containerName="authorino" Jul 07 19:12:50.449783 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:50.449657 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="488f97d9-bb43-4465-9f05-e34efab6a2f0" containerName="authorino" Jul 07 19:12:50.449783 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:50.449681 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f330977-93a1-4619-a882-78d6bfbcb8bf" containerName="authorino" Jul 07 19:12:50.449783 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:50.449690 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f330977-93a1-4619-a882-78d6bfbcb8bf" containerName="authorino" Jul 07 19:12:50.449783 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:50.449783 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f330977-93a1-4619-a882-78d6bfbcb8bf" containerName="authorino" Jul 07 19:12:50.450074 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:50.449799 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="488f97d9-bb43-4465-9f05-e34efab6a2f0" containerName="authorino" Jul 07 19:12:50.452795 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:50.452766 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c6b94b488-qmmng" Jul 07 19:12:50.460580 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:50.460554 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c6b94b488-qmmng"] Jul 07 19:12:50.563104 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:50.563072 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfkms\" (UniqueName: \"kubernetes.io/projected/8f977d89-43c2-4cfa-81d2-51879ffe4992-kube-api-access-jfkms\") pod \"authorino-6c6b94b488-qmmng\" (UID: \"8f977d89-43c2-4cfa-81d2-51879ffe4992\") " pod="kuadrant-system/authorino-6c6b94b488-qmmng" Jul 07 19:12:50.563259 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:50.563192 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f977d89-43c2-4cfa-81d2-51879ffe4992-tls-cert\") pod \"authorino-6c6b94b488-qmmng\" (UID: \"8f977d89-43c2-4cfa-81d2-51879ffe4992\") " pod="kuadrant-system/authorino-6c6b94b488-qmmng" Jul 07 19:12:50.664265 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:50.664226 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f977d89-43c2-4cfa-81d2-51879ffe4992-tls-cert\") pod \"authorino-6c6b94b488-qmmng\" (UID: \"8f977d89-43c2-4cfa-81d2-51879ffe4992\") " pod="kuadrant-system/authorino-6c6b94b488-qmmng" Jul 07 19:12:50.664423 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:50.664300 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfkms\" (UniqueName: \"kubernetes.io/projected/8f977d89-43c2-4cfa-81d2-51879ffe4992-kube-api-access-jfkms\") pod \"authorino-6c6b94b488-qmmng\" (UID: \"8f977d89-43c2-4cfa-81d2-51879ffe4992\") " pod="kuadrant-system/authorino-6c6b94b488-qmmng" Jul 07 19:12:50.666759 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:50.666724 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8f977d89-43c2-4cfa-81d2-51879ffe4992-tls-cert\") pod \"authorino-6c6b94b488-qmmng\" (UID: \"8f977d89-43c2-4cfa-81d2-51879ffe4992\") " pod="kuadrant-system/authorino-6c6b94b488-qmmng" Jul 07 19:12:50.673943 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:50.673910 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfkms\" (UniqueName: \"kubernetes.io/projected/8f977d89-43c2-4cfa-81d2-51879ffe4992-kube-api-access-jfkms\") pod \"authorino-6c6b94b488-qmmng\" (UID: \"8f977d89-43c2-4cfa-81d2-51879ffe4992\") " pod="kuadrant-system/authorino-6c6b94b488-qmmng" Jul 07 19:12:50.762955 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:50.762867 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c6b94b488-qmmng" Jul 07 19:12:50.897010 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:50.896974 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c6b94b488-qmmng"] Jul 07 19:12:50.898904 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:12:50.898872 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f977d89_43c2_4cfa_81d2_51879ffe4992.slice/crio-63ff07b45d9f7296c75cd9d90237f035c7fe2afc2e1bd1bb2a45859f5ed0bed4 WatchSource:0}: Error finding container 63ff07b45d9f7296c75cd9d90237f035c7fe2afc2e1bd1bb2a45859f5ed0bed4: Status 404 returned error can't find the container with id 63ff07b45d9f7296c75cd9d90237f035c7fe2afc2e1bd1bb2a45859f5ed0bed4 Jul 07 19:12:50.900844 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:50.900825 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:12:50.962495 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:50.962458 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c6b94b488-qmmng" event={"ID":"8f977d89-43c2-4cfa-81d2-51879ffe4992","Type":"ContainerStarted","Data":"63ff07b45d9f7296c75cd9d90237f035c7fe2afc2e1bd1bb2a45859f5ed0bed4"} Jul 07 19:12:51.971406 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:51.971362 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c6b94b488-qmmng" event={"ID":"8f977d89-43c2-4cfa-81d2-51879ffe4992","Type":"ContainerStarted","Data":"c090534e350a044518a86063b1305cf487f4fc8567f54b11c8438dd7080511d6"} Jul 07 19:12:51.988767 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:51.988701 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6c6b94b488-qmmng" podStartSLOduration=1.548456617 podStartE2EDuration="1.988679216s" totalImagesPullingTime="440.222599ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:12:51.988501053 +0000 UTC m=+742.181041144" watchObservedRunningTime="2026-07-07 19:12:51.988679216 +0000 UTC m=+742.181219284" Jul 07 19:12:52.025211 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:52.025161 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75c97d878f-mp27s"] Jul 07 19:12:52.025489 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:52.025452 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75c97d878f-mp27s" podUID="5c8f9f03-4966-4237-91ef-245fbaadc7f4" containerName="authorino" containerID="cri-o://9dec83533c0c56adf7b86f9ba114e9fbd15991b36dc06eca0eca4dc73ecd573f" gracePeriod=30 Jul 07 19:12:52.280676 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:52.280645 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75c97d878f-mp27s" Jul 07 19:12:52.381093 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:52.381062 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5c8f9f03-4966-4237-91ef-245fbaadc7f4-tls-cert\") pod \"5c8f9f03-4966-4237-91ef-245fbaadc7f4\" (UID: \"5c8f9f03-4966-4237-91ef-245fbaadc7f4\") " Jul 07 19:12:52.381281 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:52.381115 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dztpd\" (UniqueName: \"kubernetes.io/projected/5c8f9f03-4966-4237-91ef-245fbaadc7f4-kube-api-access-dztpd\") pod \"5c8f9f03-4966-4237-91ef-245fbaadc7f4\" (UID: \"5c8f9f03-4966-4237-91ef-245fbaadc7f4\") " Jul 07 19:12:52.383217 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:52.383187 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8f9f03-4966-4237-91ef-245fbaadc7f4-kube-api-access-dztpd" (OuterVolumeSpecName: "kube-api-access-dztpd") pod "5c8f9f03-4966-4237-91ef-245fbaadc7f4" (UID: "5c8f9f03-4966-4237-91ef-245fbaadc7f4"). InnerVolumeSpecName "kube-api-access-dztpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:12:52.391660 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:52.391627 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c8f9f03-4966-4237-91ef-245fbaadc7f4-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "5c8f9f03-4966-4237-91ef-245fbaadc7f4" (UID: "5c8f9f03-4966-4237-91ef-245fbaadc7f4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:12:52.482175 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:52.482145 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dztpd\" (UniqueName: \"kubernetes.io/projected/5c8f9f03-4966-4237-91ef-245fbaadc7f4-kube-api-access-dztpd\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:12:52.482175 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:52.482172 2568 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5c8f9f03-4966-4237-91ef-245fbaadc7f4-tls-cert\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:12:52.979068 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:52.978961 2568 generic.go:356] "Generic (PLEG): container finished" podID="5c8f9f03-4966-4237-91ef-245fbaadc7f4" containerID="9dec83533c0c56adf7b86f9ba114e9fbd15991b36dc06eca0eca4dc73ecd573f" exitCode=0 Jul 07 19:12:52.979068 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:52.979024 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75c97d878f-mp27s" Jul 07 19:12:52.979506 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:52.979036 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75c97d878f-mp27s" event={"ID":"5c8f9f03-4966-4237-91ef-245fbaadc7f4","Type":"ContainerDied","Data":"9dec83533c0c56adf7b86f9ba114e9fbd15991b36dc06eca0eca4dc73ecd573f"} Jul 07 19:12:52.979506 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:52.979111 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75c97d878f-mp27s" event={"ID":"5c8f9f03-4966-4237-91ef-245fbaadc7f4","Type":"ContainerDied","Data":"e0dbccae3938c880631ce70c4b91ac0c628321320848515c62d3a4f80009ad22"} Jul 07 19:12:52.979506 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:52.979129 2568 scope.go:117] "RemoveContainer" containerID="9dec83533c0c56adf7b86f9ba114e9fbd15991b36dc06eca0eca4dc73ecd573f" Jul 07 19:12:52.988412 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:52.988393 2568 scope.go:117] "RemoveContainer" containerID="9dec83533c0c56adf7b86f9ba114e9fbd15991b36dc06eca0eca4dc73ecd573f" Jul 07 19:12:52.988712 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:12:52.988680 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dec83533c0c56adf7b86f9ba114e9fbd15991b36dc06eca0eca4dc73ecd573f\": container with ID starting with 9dec83533c0c56adf7b86f9ba114e9fbd15991b36dc06eca0eca4dc73ecd573f not found: ID does not exist" containerID="9dec83533c0c56adf7b86f9ba114e9fbd15991b36dc06eca0eca4dc73ecd573f" Jul 07 19:12:52.988783 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:52.988716 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dec83533c0c56adf7b86f9ba114e9fbd15991b36dc06eca0eca4dc73ecd573f"} err="failed to get container status \"9dec83533c0c56adf7b86f9ba114e9fbd15991b36dc06eca0eca4dc73ecd573f\": rpc error: code = NotFound desc = could not find container \"9dec83533c0c56adf7b86f9ba114e9fbd15991b36dc06eca0eca4dc73ecd573f\": container with ID starting with 9dec83533c0c56adf7b86f9ba114e9fbd15991b36dc06eca0eca4dc73ecd573f not found: ID does not exist" Jul 07 19:12:53.010989 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:53.010960 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75c97d878f-mp27s"] Jul 07 19:12:53.020033 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:53.020003 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75c97d878f-mp27s"] Jul 07 19:12:54.329820 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:12:54.329786 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c8f9f03-4966-4237-91ef-245fbaadc7f4" path="/var/lib/kubelet/pods/5c8f9f03-4966-4237-91ef-245fbaadc7f4/volumes" Jul 07 19:13:19.672774 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:13:19.672730 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:13:29.360896 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:13:29.360860 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:13:40.160512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:13:40.160468 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:13:49.163810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:13:49.163762 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:14:00.154665 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:14:00.154578 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:14:26.259288 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:14:26.259242 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:14:37.472502 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:14:37.472464 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:15:13.882325 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:15:13.882287 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:15:23.465714 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:15:23.465631 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:15:31.485454 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:15:31.485353 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:15:31.493011 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:15:31.485668 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:15:31.762971 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:15:31.762892 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:15:41.964077 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:15:41.964026 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:15:51.681292 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:15:51.681253 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:16:00.676208 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:16:00.676169 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:16:12.564854 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:16:12.564815 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:16:21.563331 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:16:21.563291 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:16:30.260210 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:16:30.260178 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:16:39.876137 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:16:39.876097 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:16:51.661706 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:16:51.661666 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:17:00.164442 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:17:00.164404 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:17:25.675623 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:17:25.675583 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:17:35.977637 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:17:35.977600 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:17:49.289900 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:17:49.289856 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:17:59.403098 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:17:59.403025 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:18:10.173680 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:18:10.173642 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:18:19.170102 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:18:19.170062 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:18:27.867339 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:18:27.867258 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:20:17.080903 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:20:17.080802 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:20:26.268157 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:20:26.268104 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:20:31.523514 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:20:31.523401 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:20:31.526744 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:20:31.525466 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:20:42.960460 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:20:42.960422 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:20:53.457220 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:20:53.457185 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:21:04.754445 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:21:04.754409 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:21:14.469764 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:21:14.469728 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:21:23.381536 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:21:23.381442 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:21:34.366759 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:21:34.366723 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:21:46.274117 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:21:46.274070 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:21:54.764981 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:21:54.764942 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:22:04.964130 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:22:04.964078 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:22:30.062358 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:22:30.062325 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:22:36.768994 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:22:36.768950 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:22:46.972907 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:22:46.972869 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:22:57.678282 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:22:57.678200 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:23:07.161853 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:23:07.161816 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:23:18.209934 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:23:18.209899 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:23:26.859230 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:23:26.859190 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:23:35.484023 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:23:35.483983 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:23:44.984579 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:23:44.984537 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:23:53.861537 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:23:53.861497 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:24:04.179614 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:24:04.179571 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:24:18.053266 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:24:18.053229 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:24:26.480683 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:24:26.480599 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:24:36.005979 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:24:36.005936 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:24:47.664304 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:24:47.664261 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:24:59.076015 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:24:59.075978 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:25:08.597626 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:25:08.597591 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:25:21.866573 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:25:21.866539 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:25:31.566078 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:25:31.565961 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:25:31.569052 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:25:31.568967 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:25:33.570105 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:25:33.570071 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:25:46.067504 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:25:46.067453 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:25:57.920534 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:25:57.920457 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:26:06.381799 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:26:06.381758 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:26:17.778923 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:26:17.778884 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:26:25.452231 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:26:25.452196 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:26:33.766917 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:26:33.766877 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:26:58.071456 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:26:58.071410 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:27:06.357727 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:27:06.357675 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:28:34.264850 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:28:34.264811 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:28:44.062179 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:28:44.062138 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:28:59.670245 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:28:59.670155 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:29:10.161559 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:10.161521 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:29:42.364980 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:42.364940 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:29:58.757276 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.757241 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl"] Jul 07 19:29:58.757640 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.757603 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c8f9f03-4966-4237-91ef-245fbaadc7f4" containerName="authorino" Jul 07 19:29:58.757640 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.757614 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c8f9f03-4966-4237-91ef-245fbaadc7f4" containerName="authorino" Jul 07 19:29:58.757714 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.757683 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c8f9f03-4966-4237-91ef-245fbaadc7f4" containerName="authorino" Jul 07 19:29:58.760599 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.760582 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:58.763606 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.763580 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d23e2962\"/\"default-dockercfg-gh6z6\"" Jul 07 19:29:58.763606 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.763592 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d23e2962\"/\"kube-root-ca.crt\"" Jul 07 19:29:58.763812 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.763591 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d23e2962\"/\"openshift-service-ca.crt\"" Jul 07 19:29:58.763812 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.763600 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d23e2962\"/\"e2e-lifecycle-model-d23e2962-kserve-self-signed-certs\"" Jul 07 19:29:58.769749 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.769721 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl"] Jul 07 19:29:58.871911 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.871866 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-tmp-dir\") pod \"e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:58.872147 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.871975 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-model-cache\") pod \"e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:58.872147 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.872025 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-dshm\") pod \"e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:58.872147 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.872088 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-home\") pod \"e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:58.872147 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.872139 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c96780d-1266-4489-9c58-dad18c3db900-tls-certs\") pod \"e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:58.872358 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.872206 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj5jh\" (UniqueName: \"kubernetes.io/projected/1c96780d-1266-4489-9c58-dad18c3db900-kube-api-access-cj5jh\") pod \"e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:58.973015 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.972972 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-tmp-dir\") pod \"e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:58.973232 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.973037 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-model-cache\") pod \"e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:58.973232 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.973101 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-dshm\") pod \"e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:58.973232 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.973125 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-home\") pod \"e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:58.973232 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.973153 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c96780d-1266-4489-9c58-dad18c3db900-tls-certs\") pod \"e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:58.973232 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.973184 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj5jh\" (UniqueName: \"kubernetes.io/projected/1c96780d-1266-4489-9c58-dad18c3db900-kube-api-access-cj5jh\") pod \"e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:58.973504 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.973475 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-tmp-dir\") pod \"e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:58.973563 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.973510 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-model-cache\") pod \"e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:58.973609 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.973565 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-home\") pod \"e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:58.975532 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.975501 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-dshm\") pod \"e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:58.975703 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.975681 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c96780d-1266-4489-9c58-dad18c3db900-tls-certs\") pod \"e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:58.985200 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:58.985174 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj5jh\" (UniqueName: \"kubernetes.io/projected/1c96780d-1266-4489-9c58-dad18c3db900-kube-api-access-cj5jh\") pod \"e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:59.071351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:59.071257 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:29:59.209122 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:59.209092 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl"] Jul 07 19:29:59.210868 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:29:59.210838 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c96780d_1266_4489_9c58_dad18c3db900.slice/crio-03dbb63202f7495806165f037c53891b942da836efe6723ccb9e2aaecf6ebe2d WatchSource:0}: Error finding container 03dbb63202f7495806165f037c53891b942da836efe6723ccb9e2aaecf6ebe2d: Status 404 returned error can't find the container with id 03dbb63202f7495806165f037c53891b942da836efe6723ccb9e2aaecf6ebe2d Jul 07 19:29:59.212812 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:59.212796 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:29:59.382987 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:29:59.382955 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" event={"ID":"1c96780d-1266-4489-9c58-dad18c3db900","Type":"ContainerStarted","Data":"03dbb63202f7495806165f037c53891b942da836efe6723ccb9e2aaecf6ebe2d"} Jul 07 19:30:00.389539 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:00.389494 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" event={"ID":"1c96780d-1266-4489-9c58-dad18c3db900","Type":"ContainerStarted","Data":"2ced7d656746c507036cfdd851b4e0eaea8ea60400b6d86dcc28d7b705186cc0"} Jul 07 19:30:00.412186 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:00.412128 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" podStartSLOduration=1.3612014399999999 podStartE2EDuration="2.412110929s" totalImagesPullingTime="1.050909489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:29:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:30:00.41031769 +0000 UTC m=+1770.602857757" watchObservedRunningTime="2026-07-07 19:30:00.412110929 +0000 UTC m=+1770.604650998" Jul 07 19:30:02.255726 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:02.255687 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:30:09.072157 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:09.072114 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:30:09.072563 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:09.072171 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:30:09.084917 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:09.084884 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:30:09.446701 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:09.446673 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:30:20.460615 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:20.460577 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:30:29.164474 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:29.164432 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:30:31.601427 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:31.601303 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:30:31.610324 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:31.605465 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:30:37.088707 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:37.088670 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl"] Jul 07 19:30:40.160810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:40.160764 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:30:52.108837 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.108793 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" podUID="1c96780d-1266-4489-9c58-dad18c3db900" containerName="main" containerID="cri-o://2ced7d656746c507036cfdd851b4e0eaea8ea60400b6d86dcc28d7b705186cc0" gracePeriod=45 Jul 07 19:30:52.360989 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.360933 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:30:52.461814 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.461776 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-tmp-dir\") pod \"1c96780d-1266-4489-9c58-dad18c3db900\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " Jul 07 19:30:52.462007 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.461823 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj5jh\" (UniqueName: \"kubernetes.io/projected/1c96780d-1266-4489-9c58-dad18c3db900-kube-api-access-cj5jh\") pod \"1c96780d-1266-4489-9c58-dad18c3db900\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " Jul 07 19:30:52.462007 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.461860 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c96780d-1266-4489-9c58-dad18c3db900-tls-certs\") pod \"1c96780d-1266-4489-9c58-dad18c3db900\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " Jul 07 19:30:52.462007 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.461915 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-dshm\") pod \"1c96780d-1266-4489-9c58-dad18c3db900\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " Jul 07 19:30:52.462007 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.461986 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-model-cache\") pod \"1c96780d-1266-4489-9c58-dad18c3db900\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " Jul 07 19:30:52.462246 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.462006 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-home\") pod \"1c96780d-1266-4489-9c58-dad18c3db900\" (UID: \"1c96780d-1266-4489-9c58-dad18c3db900\") " Jul 07 19:30:52.462246 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.462095 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c96780d-1266-4489-9c58-dad18c3db900" (UID: "1c96780d-1266-4489-9c58-dad18c3db900"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:30:52.462332 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.462256 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c96780d-1266-4489-9c58-dad18c3db900" (UID: "1c96780d-1266-4489-9c58-dad18c3db900"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:30:52.462382 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.462334 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-home" (OuterVolumeSpecName: "home") pod "1c96780d-1266-4489-9c58-dad18c3db900" (UID: "1c96780d-1266-4489-9c58-dad18c3db900"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:30:52.462417 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.462387 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-model-cache\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:30:52.462417 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.462407 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-tmp-dir\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:30:52.464296 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.464265 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c96780d-1266-4489-9c58-dad18c3db900-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c96780d-1266-4489-9c58-dad18c3db900" (UID: "1c96780d-1266-4489-9c58-dad18c3db900"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:30:52.464296 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.464278 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c96780d-1266-4489-9c58-dad18c3db900-kube-api-access-cj5jh" (OuterVolumeSpecName: "kube-api-access-cj5jh") pod "1c96780d-1266-4489-9c58-dad18c3db900" (UID: "1c96780d-1266-4489-9c58-dad18c3db900"). InnerVolumeSpecName "kube-api-access-cj5jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:30:52.464464 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.464351 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-dshm" (OuterVolumeSpecName: "dshm") pod "1c96780d-1266-4489-9c58-dad18c3db900" (UID: "1c96780d-1266-4489-9c58-dad18c3db900"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:30:52.563138 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.563103 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-home\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:30:52.563138 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.563133 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj5jh\" (UniqueName: \"kubernetes.io/projected/1c96780d-1266-4489-9c58-dad18c3db900-kube-api-access-cj5jh\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:30:52.563138 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.563143 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c96780d-1266-4489-9c58-dad18c3db900-tls-certs\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:30:52.563138 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.563152 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c96780d-1266-4489-9c58-dad18c3db900-dshm\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:30:52.665650 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.665614 2568 generic.go:356] "Generic (PLEG): container finished" podID="1c96780d-1266-4489-9c58-dad18c3db900" containerID="2ced7d656746c507036cfdd851b4e0eaea8ea60400b6d86dcc28d7b705186cc0" exitCode=0 Jul 07 19:30:52.665837 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.665655 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" Jul 07 19:30:52.665837 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.665695 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" event={"ID":"1c96780d-1266-4489-9c58-dad18c3db900","Type":"ContainerDied","Data":"2ced7d656746c507036cfdd851b4e0eaea8ea60400b6d86dcc28d7b705186cc0"} Jul 07 19:30:52.665837 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.665739 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl" event={"ID":"1c96780d-1266-4489-9c58-dad18c3db900","Type":"ContainerDied","Data":"03dbb63202f7495806165f037c53891b942da836efe6723ccb9e2aaecf6ebe2d"} Jul 07 19:30:52.665837 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.665756 2568 scope.go:117] "RemoveContainer" containerID="2ced7d656746c507036cfdd851b4e0eaea8ea60400b6d86dcc28d7b705186cc0" Jul 07 19:30:52.675662 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.675645 2568 scope.go:117] "RemoveContainer" containerID="2ced7d656746c507036cfdd851b4e0eaea8ea60400b6d86dcc28d7b705186cc0" Jul 07 19:30:52.675945 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:30:52.675925 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ced7d656746c507036cfdd851b4e0eaea8ea60400b6d86dcc28d7b705186cc0\": container with ID starting with 2ced7d656746c507036cfdd851b4e0eaea8ea60400b6d86dcc28d7b705186cc0 not found: ID does not exist" containerID="2ced7d656746c507036cfdd851b4e0eaea8ea60400b6d86dcc28d7b705186cc0" Jul 07 19:30:52.675993 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.675953 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ced7d656746c507036cfdd851b4e0eaea8ea60400b6d86dcc28d7b705186cc0"} err="failed to get container status \"2ced7d656746c507036cfdd851b4e0eaea8ea60400b6d86dcc28d7b705186cc0\": rpc error: code = NotFound desc = could not find container \"2ced7d656746c507036cfdd851b4e0eaea8ea60400b6d86dcc28d7b705186cc0\": container with ID starting with 2ced7d656746c507036cfdd851b4e0eaea8ea60400b6d86dcc28d7b705186cc0 not found: ID does not exist" Jul 07 19:30:52.689140 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.689113 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl"] Jul 07 19:30:52.692583 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:52.692563 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d23e2962/e2e-lifecycle-model-d23e2962-kserve-56c857b687-xl5gl"] Jul 07 19:30:54.328571 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:30:54.328528 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c96780d-1266-4489-9c58-dad18c3db900" path="/var/lib/kubelet/pods/1c96780d-1266-4489-9c58-dad18c3db900/volumes" Jul 07 19:31:17.080110 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:31:17.080074 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:31:32.352300 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:31:32.352263 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:31:43.665718 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:31:43.665683 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:32:35.948543 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:35.948503 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m"] Jul 07 19:32:35.949098 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:35.949081 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c96780d-1266-4489-9c58-dad18c3db900" containerName="main" Jul 07 19:32:35.949154 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:35.949103 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c96780d-1266-4489-9c58-dad18c3db900" containerName="main" Jul 07 19:32:35.949205 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:35.949194 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c96780d-1266-4489-9c58-dad18c3db900" containerName="main" Jul 07 19:32:35.952533 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:35.952512 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:35.954900 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:35.954876 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 19:32:35.955104 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:35.955085 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-815c5c-openshift-default-dockercfg-xwcz2\"" Jul 07 19:32:35.962941 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:35.962915 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m"] Jul 07 19:32:36.068318 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.068279 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-workload-certs\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.068496 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.068333 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-token\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.068496 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.068400 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istiod-ca-cert\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.068496 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.068474 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2n67\" (UniqueName: \"kubernetes.io/projected/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-kube-api-access-w2n67\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.068619 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.068513 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-workload-socket\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.068619 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.068533 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-data\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.068619 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.068570 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-podinfo\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.068619 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.068608 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-credential-socket\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.068769 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.068651 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-envoy\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.169124 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.169087 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-workload-socket\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.169124 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.169129 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-data\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.169376 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.169152 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-podinfo\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.169376 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.169183 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-credential-socket\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.169376 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.169214 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-envoy\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.169376 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.169254 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-workload-certs\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.169376 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.169287 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-token\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.169376 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.169314 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istiod-ca-cert\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.170027 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.169385 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2n67\" (UniqueName: \"kubernetes.io/projected/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-kube-api-access-w2n67\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.170027 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.169545 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-data\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.170027 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.169567 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-credential-socket\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.170027 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.169729 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-workload-certs\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.170027 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.169945 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-workload-socket\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.170334 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.170178 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istiod-ca-cert\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.171623 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.171597 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-envoy\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.171933 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.171912 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-podinfo\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.181981 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.181959 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-token\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.182589 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.182563 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2n67\" (UniqueName: \"kubernetes.io/projected/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-kube-api-access-w2n67\") pod \"e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.264019 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.263923 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:36.417388 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:36.417358 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m"] Jul 07 19:32:36.419441 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:32:36.419411 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b9188b_5f7d_4c32_be54_55c6e2fe1254.slice/crio-eb7e47a291dfebda3754c35d17da315e78052c7cafac76d294623c1de1b774ab WatchSource:0}: Error finding container eb7e47a291dfebda3754c35d17da315e78052c7cafac76d294623c1de1b774ab: Status 404 returned error can't find the container with id eb7e47a291dfebda3754c35d17da315e78052c7cafac76d294623c1de1b774ab Jul 07 19:32:37.207687 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:37.207648 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" event={"ID":"c7b9188b-5f7d-4c32-be54-55c6e2fe1254","Type":"ContainerStarted","Data":"eb7e47a291dfebda3754c35d17da315e78052c7cafac76d294623c1de1b774ab"} Jul 07 19:32:38.564818 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:38.564783 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:32:39.364742 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:39.364702 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 07 19:32:39.364833 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:39.364778 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 07 19:32:39.364833 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:39.364809 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 07 19:32:40.228542 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:40.228504 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" event={"ID":"c7b9188b-5f7d-4c32-be54-55c6e2fe1254","Type":"ContainerStarted","Data":"4df3bb7827cf17ee8b8027d1b19312d72814d51ad6ec544335c86a3b898ac771"} Jul 07 19:32:40.251686 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:40.251620 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" podStartSLOduration=2.309006711 podStartE2EDuration="5.251603938s" totalImagesPullingTime="2.942597227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:32:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:32:40.248724425 +0000 UTC m=+1930.441264491" watchObservedRunningTime="2026-07-07 19:32:40.251603938 +0000 UTC m=+1930.444144004" Jul 07 19:32:40.265121 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:40.265086 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:40.269887 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:40.269852 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:41.234750 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:41.234717 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:41.235832 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:41.235815 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:51.555611 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:51.555568 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:32:53.303643 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:53.303545 2568 generic.go:356] "Generic (PLEG): container finished" podID="c7b9188b-5f7d-4c32-be54-55c6e2fe1254" containerID="4df3bb7827cf17ee8b8027d1b19312d72814d51ad6ec544335c86a3b898ac771" exitCode=0 Jul 07 19:32:53.303643 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:53.303601 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" event={"ID":"c7b9188b-5f7d-4c32-be54-55c6e2fe1254","Type":"ContainerDied","Data":"4df3bb7827cf17ee8b8027d1b19312d72814d51ad6ec544335c86a3b898ac771"} Jul 07 19:32:53.304110 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:53.303970 2568 scope.go:117] "RemoveContainer" containerID="4df3bb7827cf17ee8b8027d1b19312d72814d51ad6ec544335c86a3b898ac771" Jul 07 19:32:53.306172 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:53.306133 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 07 19:32:53.306248 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:53.306219 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 07 19:32:53.306285 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:53.306253 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 07 19:32:54.310544 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:54.310503 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" event={"ID":"c7b9188b-5f7d-4c32-be54-55c6e2fe1254","Type":"ContainerStarted","Data":"c249ad9e79d8e9c6b10294925d2f2891f65d3b849d4ac7490a4b5ebb016b11e2"} Jul 07 19:32:55.264816 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:55.264769 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:55.266091 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:55.266061 2568 patch_prober.go:28] interesting pod/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 07 19:32:55.266222 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:55.266118 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" podUID="c7b9188b-5f7d-4c32-be54-55c6e2fe1254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 07 19:32:56.235342 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:56.235305 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:56.264531 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:56.264484 2568 patch_prober.go:28] interesting pod/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 07 19:32:56.264729 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:56.264553 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" podUID="c7b9188b-5f7d-4c32-be54-55c6e2fe1254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 07 19:32:57.264727 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:57.264682 2568 patch_prober.go:28] interesting pod/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 07 19:32:57.265144 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:57.264766 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" podUID="c7b9188b-5f7d-4c32-be54-55c6e2fe1254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 07 19:32:58.265218 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:58.265175 2568 patch_prober.go:28] interesting pod/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 07 19:32:58.265614 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:58.265246 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" podUID="c7b9188b-5f7d-4c32-be54-55c6e2fe1254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 07 19:32:59.268934 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:59.268893 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:32:59.269992 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:32:59.269973 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:33:11.169338 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:33:11.169302 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:33:33.356828 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:33:33.356793 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:33:46.460701 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:33:46.460661 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:34:02.259938 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:02.259890 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:34:39.199191 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:39.199151 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:34:42.279109 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:42.279037 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m"] Jul 07 19:34:42.279493 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:42.279329 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" podUID="c7b9188b-5f7d-4c32-be54-55c6e2fe1254" containerName="istio-proxy" containerID="cri-o://c249ad9e79d8e9c6b10294925d2f2891f65d3b849d4ac7490a4b5ebb016b11e2" gracePeriod=30 Jul 07 19:34:44.271951 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:44.271907 2568 patch_prober.go:28] interesting pod/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 19:34:44.272351 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:44.271969 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" podUID="c7b9188b-5f7d-4c32-be54-55c6e2fe1254" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 19:34:47.537572 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.537541 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:34:47.650668 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.650634 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istiod-ca-cert\") pod \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " Jul 07 19:34:47.650860 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.650678 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-workload-socket\") pod \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " Jul 07 19:34:47.650860 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.650712 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-token\") pod \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " Jul 07 19:34:47.650860 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.650736 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-data\") pod \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " Jul 07 19:34:47.650860 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.650772 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-credential-socket\") pod \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " Jul 07 19:34:47.650860 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.650798 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2n67\" (UniqueName: \"kubernetes.io/projected/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-kube-api-access-w2n67\") pod \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " Jul 07 19:34:47.650860 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.650824 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-podinfo\") pod \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " Jul 07 19:34:47.651198 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.650967 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c7b9188b-5f7d-4c32-be54-55c6e2fe1254" (UID: "c7b9188b-5f7d-4c32-be54-55c6e2fe1254"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:47.651198 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.650994 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-workload-certs\") pod \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " Jul 07 19:34:47.651198 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.651088 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-envoy\") pod \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\" (UID: \"c7b9188b-5f7d-4c32-be54-55c6e2fe1254\") " Jul 07 19:34:47.651198 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.651124 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c7b9188b-5f7d-4c32-be54-55c6e2fe1254" (UID: "c7b9188b-5f7d-4c32-be54-55c6e2fe1254"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:34:47.651411 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.651290 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c7b9188b-5f7d-4c32-be54-55c6e2fe1254" (UID: "c7b9188b-5f7d-4c32-be54-55c6e2fe1254"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:47.651411 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.651318 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c7b9188b-5f7d-4c32-be54-55c6e2fe1254" (UID: "c7b9188b-5f7d-4c32-be54-55c6e2fe1254"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:47.651515 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.651415 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-workload-certs\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:34:47.651515 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.651433 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istiod-ca-cert\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:34:47.651515 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.651446 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-workload-socket\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:34:47.651515 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.651460 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-credential-socket\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:34:47.651855 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.651829 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-data" (OuterVolumeSpecName: "istio-data") pod "c7b9188b-5f7d-4c32-be54-55c6e2fe1254" (UID: "c7b9188b-5f7d-4c32-be54-55c6e2fe1254"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:47.653391 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.653365 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c7b9188b-5f7d-4c32-be54-55c6e2fe1254" (UID: "c7b9188b-5f7d-4c32-be54-55c6e2fe1254"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:34:47.653501 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.653418 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-kube-api-access-w2n67" (OuterVolumeSpecName: "kube-api-access-w2n67") pod "c7b9188b-5f7d-4c32-be54-55c6e2fe1254" (UID: "c7b9188b-5f7d-4c32-be54-55c6e2fe1254"). InnerVolumeSpecName "kube-api-access-w2n67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:34:47.653501 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.653429 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-token" (OuterVolumeSpecName: "istio-token") pod "c7b9188b-5f7d-4c32-be54-55c6e2fe1254" (UID: "c7b9188b-5f7d-4c32-be54-55c6e2fe1254"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:34:47.653501 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.653451 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c7b9188b-5f7d-4c32-be54-55c6e2fe1254" (UID: "c7b9188b-5f7d-4c32-be54-55c6e2fe1254"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:34:47.752618 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.752579 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-envoy\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:34:47.752618 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.752609 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-token\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:34:47.752618 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.752618 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-data\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:34:47.752849 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.752631 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2n67\" (UniqueName: \"kubernetes.io/projected/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-kube-api-access-w2n67\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:34:47.752849 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.752645 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b9188b-5f7d-4c32-be54-55c6e2fe1254-istio-podinfo\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:34:47.919453 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.919354 2568 generic.go:356] "Generic (PLEG): container finished" podID="c7b9188b-5f7d-4c32-be54-55c6e2fe1254" containerID="c249ad9e79d8e9c6b10294925d2f2891f65d3b849d4ac7490a4b5ebb016b11e2" exitCode=0 Jul 07 19:34:47.919453 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.919413 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" Jul 07 19:34:47.919695 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.919449 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" event={"ID":"c7b9188b-5f7d-4c32-be54-55c6e2fe1254","Type":"ContainerDied","Data":"c249ad9e79d8e9c6b10294925d2f2891f65d3b849d4ac7490a4b5ebb016b11e2"} Jul 07 19:34:47.919695 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.919519 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m" event={"ID":"c7b9188b-5f7d-4c32-be54-55c6e2fe1254","Type":"ContainerDied","Data":"eb7e47a291dfebda3754c35d17da315e78052c7cafac76d294623c1de1b774ab"} Jul 07 19:34:47.919695 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.919549 2568 scope.go:117] "RemoveContainer" containerID="c249ad9e79d8e9c6b10294925d2f2891f65d3b849d4ac7490a4b5ebb016b11e2" Jul 07 19:34:47.931415 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.931394 2568 scope.go:117] "RemoveContainer" containerID="4df3bb7827cf17ee8b8027d1b19312d72814d51ad6ec544335c86a3b898ac771" Jul 07 19:34:47.940139 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.940113 2568 scope.go:117] "RemoveContainer" containerID="c249ad9e79d8e9c6b10294925d2f2891f65d3b849d4ac7490a4b5ebb016b11e2" Jul 07 19:34:47.940452 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:34:47.940424 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c249ad9e79d8e9c6b10294925d2f2891f65d3b849d4ac7490a4b5ebb016b11e2\": container with ID starting with c249ad9e79d8e9c6b10294925d2f2891f65d3b849d4ac7490a4b5ebb016b11e2 not found: ID does not exist" containerID="c249ad9e79d8e9c6b10294925d2f2891f65d3b849d4ac7490a4b5ebb016b11e2" Jul 07 19:34:47.940531 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.940453 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c249ad9e79d8e9c6b10294925d2f2891f65d3b849d4ac7490a4b5ebb016b11e2"} err="failed to get container status \"c249ad9e79d8e9c6b10294925d2f2891f65d3b849d4ac7490a4b5ebb016b11e2\": rpc error: code = NotFound desc = could not find container \"c249ad9e79d8e9c6b10294925d2f2891f65d3b849d4ac7490a4b5ebb016b11e2\": container with ID starting with c249ad9e79d8e9c6b10294925d2f2891f65d3b849d4ac7490a4b5ebb016b11e2 not found: ID does not exist" Jul 07 19:34:47.940531 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.940470 2568 scope.go:117] "RemoveContainer" containerID="4df3bb7827cf17ee8b8027d1b19312d72814d51ad6ec544335c86a3b898ac771" Jul 07 19:34:47.940737 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:34:47.940716 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4df3bb7827cf17ee8b8027d1b19312d72814d51ad6ec544335c86a3b898ac771\": container with ID starting with 4df3bb7827cf17ee8b8027d1b19312d72814d51ad6ec544335c86a3b898ac771 not found: ID does not exist" containerID="4df3bb7827cf17ee8b8027d1b19312d72814d51ad6ec544335c86a3b898ac771" Jul 07 19:34:47.940779 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.940744 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4df3bb7827cf17ee8b8027d1b19312d72814d51ad6ec544335c86a3b898ac771"} err="failed to get container status \"4df3bb7827cf17ee8b8027d1b19312d72814d51ad6ec544335c86a3b898ac771\": rpc error: code = NotFound desc = could not find container \"4df3bb7827cf17ee8b8027d1b19312d72814d51ad6ec544335c86a3b898ac771\": container with ID starting with 4df3bb7827cf17ee8b8027d1b19312d72814d51ad6ec544335c86a3b898ac771 not found: ID does not exist" Jul 07 19:34:47.950719 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.950695 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m"] Jul 07 19:34:47.956268 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:47.956243 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-815c5c-openshift-default-74cdf477c7-nv59m"] Jul 07 19:34:48.328622 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:48.328541 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7b9188b-5f7d-4c32-be54-55c6e2fe1254" path="/var/lib/kubelet/pods/c7b9188b-5f7d-4c32-be54-55c6e2fe1254/volumes" Jul 07 19:34:50.659756 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:34:50.659717 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:35:05.070782 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:05.070703 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:35:18.759245 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:18.759204 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:35:23.416941 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.416899 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf"] Jul 07 19:35:23.417477 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.417456 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7b9188b-5f7d-4c32-be54-55c6e2fe1254" containerName="istio-proxy" Jul 07 19:35:23.417556 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.417482 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7b9188b-5f7d-4c32-be54-55c6e2fe1254" containerName="istio-proxy" Jul 07 19:35:23.417556 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.417514 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7b9188b-5f7d-4c32-be54-55c6e2fe1254" containerName="istio-proxy" Jul 07 19:35:23.417556 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.417523 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7b9188b-5f7d-4c32-be54-55c6e2fe1254" containerName="istio-proxy" Jul 07 19:35:23.417711 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.417613 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7b9188b-5f7d-4c32-be54-55c6e2fe1254" containerName="istio-proxy" Jul 07 19:35:23.420729 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.420704 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.424215 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.424191 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 19:35:23.424343 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.424225 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-0e4903-openshift-default-dockercfg-jkvg6\"" Jul 07 19:35:23.437593 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.437562 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf"] Jul 07 19:35:23.472168 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.472127 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-workload-certs\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.472348 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.472176 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-workload-socket\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.472348 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.472196 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-envoy\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.472348 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.472254 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d474521-6484-42c0-a045-a36dd5a7d49f-istiod-ca-cert\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.472348 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.472303 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-podinfo\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.472489 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.472367 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-data\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.472489 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.472405 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j2tg\" (UniqueName: \"kubernetes.io/projected/2d474521-6484-42c0-a045-a36dd5a7d49f-kube-api-access-7j2tg\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.472489 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.472460 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-token\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.472587 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.472515 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-credential-socket\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.573172 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.573125 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-credential-socket\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.573172 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.573176 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-workload-certs\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.573402 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.573314 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-workload-socket\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.573402 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.573366 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-envoy\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.573402 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.573393 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d474521-6484-42c0-a045-a36dd5a7d49f-istiod-ca-cert\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.573554 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.573423 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-podinfo\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.573554 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.573473 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-data\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.573664 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.573594 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-workload-certs\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.573664 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.573601 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j2tg\" (UniqueName: \"kubernetes.io/projected/2d474521-6484-42c0-a045-a36dd5a7d49f-kube-api-access-7j2tg\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.573664 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.573642 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-credential-socket\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.573812 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.573684 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-token\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.573861 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.573842 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-data\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.573993 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.573965 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-workload-socket\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.574417 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.574390 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d474521-6484-42c0-a045-a36dd5a7d49f-istiod-ca-cert\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.575729 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.575707 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-envoy\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.576029 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.576010 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-podinfo\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.589594 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.589555 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-token\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.589741 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.589651 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j2tg\" (UniqueName: \"kubernetes.io/projected/2d474521-6484-42c0-a045-a36dd5a7d49f-kube-api-access-7j2tg\") pod \"e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.731209 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.731099 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:23.881424 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.881395 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf"] Jul 07 19:35:23.883012 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:35:23.882979 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d474521_6484_42c0_a045_a36dd5a7d49f.slice/crio-96cdcc8396622d80ad69c06e51cb4bc00862d0309c5868a8f6eb9402ddf821f2 WatchSource:0}: Error finding container 96cdcc8396622d80ad69c06e51cb4bc00862d0309c5868a8f6eb9402ddf821f2: Status 404 returned error can't find the container with id 96cdcc8396622d80ad69c06e51cb4bc00862d0309c5868a8f6eb9402ddf821f2 Jul 07 19:35:23.885252 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.885211 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 07 19:35:23.885325 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.885290 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 07 19:35:23.885363 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:23.885327 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 07 19:35:24.109087 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:24.108975 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" event={"ID":"2d474521-6484-42c0-a045-a36dd5a7d49f","Type":"ContainerStarted","Data":"98c8d1bb851cae6d8473127594248e15a5bb6bc60976b08ab7cf0989db9653a0"} Jul 07 19:35:24.109087 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:24.109064 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" event={"ID":"2d474521-6484-42c0-a045-a36dd5a7d49f","Type":"ContainerStarted","Data":"96cdcc8396622d80ad69c06e51cb4bc00862d0309c5868a8f6eb9402ddf821f2"} Jul 07 19:35:24.132680 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:24.132627 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" podStartSLOduration=1.132611516 podStartE2EDuration="1.132611516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:35:24.130963616 +0000 UTC m=+2094.323503684" watchObservedRunningTime="2026-07-07 19:35:24.132611516 +0000 UTC m=+2094.325151582" Jul 07 19:35:24.732117 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:24.732081 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:24.737003 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:24.736976 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:25.114240 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:25.114151 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:25.115104 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:25.115086 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:31.638232 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:31.638204 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:35:31.646749 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:31.646725 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:35:31.957165 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:31.957080 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:35:40.116135 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:40.116022 2568 patch_prober.go:28] interesting pod/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 07 19:35:40.116518 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:40.116125 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" podUID="2d474521-6484-42c0-a045-a36dd5a7d49f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 07 19:35:40.208905 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:40.208861 2568 generic.go:356] "Generic (PLEG): container finished" podID="2d474521-6484-42c0-a045-a36dd5a7d49f" containerID="98c8d1bb851cae6d8473127594248e15a5bb6bc60976b08ab7cf0989db9653a0" exitCode=0 Jul 07 19:35:40.208905 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:40.208909 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" event={"ID":"2d474521-6484-42c0-a045-a36dd5a7d49f","Type":"ContainerDied","Data":"98c8d1bb851cae6d8473127594248e15a5bb6bc60976b08ab7cf0989db9653a0"} Jul 07 19:35:40.209315 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:40.209300 2568 scope.go:117] "RemoveContainer" containerID="98c8d1bb851cae6d8473127594248e15a5bb6bc60976b08ab7cf0989db9653a0" Jul 07 19:35:40.210297 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:40.210267 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 07 19:35:40.210354 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:40.210336 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 07 19:35:40.210390 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:40.210366 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 07 19:35:41.215713 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.215678 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" event={"ID":"2d474521-6484-42c0-a045-a36dd5a7d49f","Type":"ContainerStarted","Data":"fb89cafc032784d9f311769aa81e67479fd587ccb0fb10fd66fc51913158ebb8"} Jul 07 19:35:41.731834 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.731791 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:41.732155 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.732015 2568 patch_prober.go:28] interesting pod/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 07 19:35:41.732362 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.732330 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" podUID="2d474521-6484-42c0-a045-a36dd5a7d49f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 07 19:35:41.766070 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.765265 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj"] Jul 07 19:35:41.769237 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.769204 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7b9188b-5f7d-4c32-be54-55c6e2fe1254" containerName="istio-proxy" Jul 07 19:35:41.771987 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.771962 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:41.774436 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.774402 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0e4903\"/\"openshift-service-ca.crt\"" Jul 07 19:35:41.774939 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.774918 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0e4903\"/\"kube-root-ca.crt\"" Jul 07 19:35:41.775007 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.774973 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0e4903\"/\"auth-test-model-0e4903-kserve-self-signed-certs\"" Jul 07 19:35:41.775164 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.775145 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0e4903\"/\"default-dockercfg-b9fbb\"" Jul 07 19:35:41.779750 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.779720 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj"] Jul 07 19:35:41.858071 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.857397 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-home\") pod \"auth-test-model-0e4903-kserve-58844897bc-s8gxj\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:41.858071 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.857489 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nddt\" (UniqueName: \"kubernetes.io/projected/8a225eb5-10f7-4638-86f6-fb0d7208c70c-kube-api-access-7nddt\") pod \"auth-test-model-0e4903-kserve-58844897bc-s8gxj\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:41.858071 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.857535 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a225eb5-10f7-4638-86f6-fb0d7208c70c-tls-certs\") pod \"auth-test-model-0e4903-kserve-58844897bc-s8gxj\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:41.858071 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.857601 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-model-cache\") pod \"auth-test-model-0e4903-kserve-58844897bc-s8gxj\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:41.858071 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.857624 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-tmp-dir\") pod \"auth-test-model-0e4903-kserve-58844897bc-s8gxj\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:41.858071 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.857674 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-dshm\") pod \"auth-test-model-0e4903-kserve-58844897bc-s8gxj\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:41.958619 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.958569 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-model-cache\") pod \"auth-test-model-0e4903-kserve-58844897bc-s8gxj\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:41.958619 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.958627 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-tmp-dir\") pod \"auth-test-model-0e4903-kserve-58844897bc-s8gxj\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:41.958895 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.958674 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-dshm\") pod \"auth-test-model-0e4903-kserve-58844897bc-s8gxj\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:41.958895 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.958712 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-home\") pod \"auth-test-model-0e4903-kserve-58844897bc-s8gxj\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:41.958895 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.958785 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nddt\" (UniqueName: \"kubernetes.io/projected/8a225eb5-10f7-4638-86f6-fb0d7208c70c-kube-api-access-7nddt\") pod \"auth-test-model-0e4903-kserve-58844897bc-s8gxj\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:41.958895 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.958827 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a225eb5-10f7-4638-86f6-fb0d7208c70c-tls-certs\") pod \"auth-test-model-0e4903-kserve-58844897bc-s8gxj\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:41.959912 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.959882 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-model-cache\") pod \"auth-test-model-0e4903-kserve-58844897bc-s8gxj\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:41.960027 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.959922 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-tmp-dir\") pod \"auth-test-model-0e4903-kserve-58844897bc-s8gxj\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:41.960112 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.960056 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-home\") pod \"auth-test-model-0e4903-kserve-58844897bc-s8gxj\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:41.963072 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.962758 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a225eb5-10f7-4638-86f6-fb0d7208c70c-tls-certs\") pod \"auth-test-model-0e4903-kserve-58844897bc-s8gxj\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:41.963972 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.963923 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-dshm\") pod \"auth-test-model-0e4903-kserve-58844897bc-s8gxj\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:41.971139 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:41.971115 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nddt\" (UniqueName: \"kubernetes.io/projected/8a225eb5-10f7-4638-86f6-fb0d7208c70c-kube-api-access-7nddt\") pod \"auth-test-model-0e4903-kserve-58844897bc-s8gxj\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:42.084020 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:42.083921 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:42.265305 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:42.265263 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj"] Jul 07 19:35:42.270640 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:35:42.270609 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a225eb5_10f7_4638_86f6_fb0d7208c70c.slice/crio-aa8691a78bc6836eca0c93cb99f7e3e82fca16dbaa78937ec303c83cc96c4eaa WatchSource:0}: Error finding container aa8691a78bc6836eca0c93cb99f7e3e82fca16dbaa78937ec303c83cc96c4eaa: Status 404 returned error can't find the container with id aa8691a78bc6836eca0c93cb99f7e3e82fca16dbaa78937ec303c83cc96c4eaa Jul 07 19:35:42.731803 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:42.731764 2568 patch_prober.go:28] interesting pod/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 07 19:35:42.731982 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:42.731821 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" podUID="2d474521-6484-42c0-a045-a36dd5a7d49f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 07 19:35:43.230211 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:43.230170 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" event={"ID":"8a225eb5-10f7-4638-86f6-fb0d7208c70c","Type":"ContainerStarted","Data":"ece771f36f25f387552df6f48149afce5218220c29fbc86fb8165b7511a87b72"} Jul 07 19:35:43.230211 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:43.230218 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" event={"ID":"8a225eb5-10f7-4638-86f6-fb0d7208c70c","Type":"ContainerStarted","Data":"aa8691a78bc6836eca0c93cb99f7e3e82fca16dbaa78937ec303c83cc96c4eaa"} Jul 07 19:35:43.732321 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:43.732282 2568 patch_prober.go:28] interesting pod/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 07 19:35:43.732726 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:43.732341 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" podUID="2d474521-6484-42c0-a045-a36dd5a7d49f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 07 19:35:44.731668 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:44.731628 2568 patch_prober.go:28] interesting pod/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 07 19:35:44.731844 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:44.731686 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" podUID="2d474521-6484-42c0-a045-a36dd5a7d49f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 07 19:35:44.752783 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:44.752701 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" podStartSLOduration=3.75267925 podStartE2EDuration="3.75267925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:35:43.252137099 +0000 UTC m=+2113.444677181" watchObservedRunningTime="2026-07-07 19:35:44.75267925 +0000 UTC m=+2114.945219318" Jul 07 19:35:44.757840 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:44.757802 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:35:45.736330 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:45.736298 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:45.736596 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:45.736563 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:45.737374 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:45.737356 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:35:52.084686 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:52.084650 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:52.085081 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:52.084708 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:52.097001 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:52.096978 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:35:52.290444 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:35:52.290414 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:36:14.270862 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:36:14.270820 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:36:31.270992 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:36:31.270898 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:36:45.364917 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:36:45.364869 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:37:01.850233 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:37:01.850202 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:37:17.657762 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:37:17.657720 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:37:34.068306 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:37:34.068269 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:37:50.557574 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:37:50.557536 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:38:06.265661 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:06.265577 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:38:09.207253 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:09.207220 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj"] Jul 07 19:38:22.097398 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:22.097363 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:38:24.227285 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.227220 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" podUID="8a225eb5-10f7-4638-86f6-fb0d7208c70c" containerName="main" containerID="cri-o://ece771f36f25f387552df6f48149afce5218220c29fbc86fb8165b7511a87b72" gracePeriod=45 Jul 07 19:38:24.470275 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.470252 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:38:24.544807 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.544724 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-home\") pod \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " Jul 07 19:38:24.544807 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.544761 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nddt\" (UniqueName: \"kubernetes.io/projected/8a225eb5-10f7-4638-86f6-fb0d7208c70c-kube-api-access-7nddt\") pod \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " Jul 07 19:38:24.544807 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.544800 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-model-cache\") pod \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " Jul 07 19:38:24.545143 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.544838 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-dshm\") pod \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " Jul 07 19:38:24.545143 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.544863 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a225eb5-10f7-4638-86f6-fb0d7208c70c-tls-certs\") pod \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " Jul 07 19:38:24.545143 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.544893 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-tmp-dir\") pod \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\" (UID: \"8a225eb5-10f7-4638-86f6-fb0d7208c70c\") " Jul 07 19:38:24.545143 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.545084 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-home" (OuterVolumeSpecName: "home") pod "8a225eb5-10f7-4638-86f6-fb0d7208c70c" (UID: "8a225eb5-10f7-4638-86f6-fb0d7208c70c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:24.545352 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.545159 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a225eb5-10f7-4638-86f6-fb0d7208c70c" (UID: "8a225eb5-10f7-4638-86f6-fb0d7208c70c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:24.545352 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.545233 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a225eb5-10f7-4638-86f6-fb0d7208c70c" (UID: "8a225eb5-10f7-4638-86f6-fb0d7208c70c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:24.547207 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.547174 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a225eb5-10f7-4638-86f6-fb0d7208c70c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a225eb5-10f7-4638-86f6-fb0d7208c70c" (UID: "8a225eb5-10f7-4638-86f6-fb0d7208c70c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:38:24.547319 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.547233 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-dshm" (OuterVolumeSpecName: "dshm") pod "8a225eb5-10f7-4638-86f6-fb0d7208c70c" (UID: "8a225eb5-10f7-4638-86f6-fb0d7208c70c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:24.547319 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.547237 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a225eb5-10f7-4638-86f6-fb0d7208c70c-kube-api-access-7nddt" (OuterVolumeSpecName: "kube-api-access-7nddt") pod "8a225eb5-10f7-4638-86f6-fb0d7208c70c" (UID: "8a225eb5-10f7-4638-86f6-fb0d7208c70c"). InnerVolumeSpecName "kube-api-access-7nddt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:38:24.645569 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.645536 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a225eb5-10f7-4638-86f6-fb0d7208c70c-tls-certs\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:38:24.645569 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.645563 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-tmp-dir\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:38:24.645569 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.645572 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-home\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:38:24.645793 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.645582 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nddt\" (UniqueName: \"kubernetes.io/projected/8a225eb5-10f7-4638-86f6-fb0d7208c70c-kube-api-access-7nddt\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:38:24.645793 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.645594 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-model-cache\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:38:24.645793 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:24.645602 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a225eb5-10f7-4638-86f6-fb0d7208c70c-dshm\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:38:25.100063 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:25.100020 2568 generic.go:356] "Generic (PLEG): container finished" podID="8a225eb5-10f7-4638-86f6-fb0d7208c70c" containerID="ece771f36f25f387552df6f48149afce5218220c29fbc86fb8165b7511a87b72" exitCode=0 Jul 07 19:38:25.100245 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:25.100082 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" Jul 07 19:38:25.100245 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:25.100080 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" event={"ID":"8a225eb5-10f7-4638-86f6-fb0d7208c70c","Type":"ContainerDied","Data":"ece771f36f25f387552df6f48149afce5218220c29fbc86fb8165b7511a87b72"} Jul 07 19:38:25.100245 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:25.100137 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj" event={"ID":"8a225eb5-10f7-4638-86f6-fb0d7208c70c","Type":"ContainerDied","Data":"aa8691a78bc6836eca0c93cb99f7e3e82fca16dbaa78937ec303c83cc96c4eaa"} Jul 07 19:38:25.100245 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:25.100164 2568 scope.go:117] "RemoveContainer" containerID="ece771f36f25f387552df6f48149afce5218220c29fbc86fb8165b7511a87b72" Jul 07 19:38:25.110163 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:25.110143 2568 scope.go:117] "RemoveContainer" containerID="ece771f36f25f387552df6f48149afce5218220c29fbc86fb8165b7511a87b72" Jul 07 19:38:25.110385 ip-10-0-141-109 kubenswrapper[2568]: E0707 19:38:25.110365 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece771f36f25f387552df6f48149afce5218220c29fbc86fb8165b7511a87b72\": container with ID starting with ece771f36f25f387552df6f48149afce5218220c29fbc86fb8165b7511a87b72 not found: ID does not exist" containerID="ece771f36f25f387552df6f48149afce5218220c29fbc86fb8165b7511a87b72" Jul 07 19:38:25.110451 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:25.110393 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece771f36f25f387552df6f48149afce5218220c29fbc86fb8165b7511a87b72"} err="failed to get container status \"ece771f36f25f387552df6f48149afce5218220c29fbc86fb8165b7511a87b72\": rpc error: code = NotFound desc = could not find container \"ece771f36f25f387552df6f48149afce5218220c29fbc86fb8165b7511a87b72\": container with ID starting with ece771f36f25f387552df6f48149afce5218220c29fbc86fb8165b7511a87b72 not found: ID does not exist" Jul 07 19:38:25.124318 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:25.124294 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj"] Jul 07 19:38:25.129403 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:25.129383 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0e4903/auth-test-model-0e4903-kserve-58844897bc-s8gxj"] Jul 07 19:38:26.329167 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:26.329130 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a225eb5-10f7-4638-86f6-fb0d7208c70c" path="/var/lib/kubelet/pods/8a225eb5-10f7-4638-86f6-fb0d7208c70c/volumes" Jul 07 19:38:42.054471 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:38:42.054430 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:39:04.376180 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:39:04.376138 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:39:18.164495 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:39:18.164452 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:39:38.196558 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:39:38.196461 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:39:59.455964 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:39:59.455930 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:40:24.555438 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:40:24.555389 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:40:31.677951 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:40:31.677921 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:40:31.688537 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:40:31.688513 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sstqk_8e2c0f4e-48ec-4049-bdff-1e9b8052f324/ovn-acl-logging/0.log" Jul 07 19:40:38.862721 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:40:38.862681 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:40:53.965323 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:40:53.965250 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:41:08.366917 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:08.366877 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:41:48.053874 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:48.053832 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf"] Jul 07 19:41:48.054481 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:48.054123 2568 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" podUID="2d474521-6484-42c0-a045-a36dd5a7d49f" containerName="istio-proxy" containerID="cri-o://fb89cafc032784d9f311769aa81e67479fd587ccb0fb10fd66fc51913158ebb8" gracePeriod=30 Jul 07 19:41:52.172685 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:52.172628 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-vmrzr"] Jul 07 19:41:53.198189 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.198155 2568 generic.go:356] "Generic (PLEG): container finished" podID="2d474521-6484-42c0-a045-a36dd5a7d49f" containerID="fb89cafc032784d9f311769aa81e67479fd587ccb0fb10fd66fc51913158ebb8" exitCode=0 Jul 07 19:41:53.198581 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.198238 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" event={"ID":"2d474521-6484-42c0-a045-a36dd5a7d49f","Type":"ContainerDied","Data":"fb89cafc032784d9f311769aa81e67479fd587ccb0fb10fd66fc51913158ebb8"} Jul 07 19:41:53.198581 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.198321 2568 scope.go:117] "RemoveContainer" containerID="98c8d1bb851cae6d8473127594248e15a5bb6bc60976b08ab7cf0989db9653a0" Jul 07 19:41:53.318038 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.317977 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:41:53.396928 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.396897 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d474521-6484-42c0-a045-a36dd5a7d49f-istiod-ca-cert\") pod \"2d474521-6484-42c0-a045-a36dd5a7d49f\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " Jul 07 19:41:53.397149 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.396962 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-podinfo\") pod \"2d474521-6484-42c0-a045-a36dd5a7d49f\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " Jul 07 19:41:53.397149 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.396997 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-envoy\") pod \"2d474521-6484-42c0-a045-a36dd5a7d49f\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " Jul 07 19:41:53.397149 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.397113 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-data\") pod \"2d474521-6484-42c0-a045-a36dd5a7d49f\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " Jul 07 19:41:53.397338 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.397156 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-workload-socket\") pod \"2d474521-6484-42c0-a045-a36dd5a7d49f\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " Jul 07 19:41:53.397338 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.397184 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j2tg\" (UniqueName: \"kubernetes.io/projected/2d474521-6484-42c0-a045-a36dd5a7d49f-kube-api-access-7j2tg\") pod \"2d474521-6484-42c0-a045-a36dd5a7d49f\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " Jul 07 19:41:53.397338 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.397210 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-token\") pod \"2d474521-6484-42c0-a045-a36dd5a7d49f\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " Jul 07 19:41:53.397338 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.397230 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-credential-socket\") pod \"2d474521-6484-42c0-a045-a36dd5a7d49f\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " Jul 07 19:41:53.397338 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.397289 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-workload-certs\") pod \"2d474521-6484-42c0-a045-a36dd5a7d49f\" (UID: \"2d474521-6484-42c0-a045-a36dd5a7d49f\") " Jul 07 19:41:53.397582 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.397394 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d474521-6484-42c0-a045-a36dd5a7d49f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2d474521-6484-42c0-a045-a36dd5a7d49f" (UID: "2d474521-6484-42c0-a045-a36dd5a7d49f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:41:53.397636 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.397598 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2d474521-6484-42c0-a045-a36dd5a7d49f" (UID: "2d474521-6484-42c0-a045-a36dd5a7d49f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:41:53.397771 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.397716 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-workload-certs\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:41:53.397771 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.397742 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d474521-6484-42c0-a045-a36dd5a7d49f-istiod-ca-cert\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:41:53.397902 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.397780 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2d474521-6484-42c0-a045-a36dd5a7d49f" (UID: "2d474521-6484-42c0-a045-a36dd5a7d49f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:41:53.397902 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.397792 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2d474521-6484-42c0-a045-a36dd5a7d49f" (UID: "2d474521-6484-42c0-a045-a36dd5a7d49f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:41:53.398384 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.398349 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-data" (OuterVolumeSpecName: "istio-data") pod "2d474521-6484-42c0-a045-a36dd5a7d49f" (UID: "2d474521-6484-42c0-a045-a36dd5a7d49f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:41:53.399494 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.399467 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2d474521-6484-42c0-a045-a36dd5a7d49f" (UID: "2d474521-6484-42c0-a045-a36dd5a7d49f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 19:41:53.399606 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.399544 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-token" (OuterVolumeSpecName: "istio-token") pod "2d474521-6484-42c0-a045-a36dd5a7d49f" (UID: "2d474521-6484-42c0-a045-a36dd5a7d49f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:41:53.399686 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.399672 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2d474521-6484-42c0-a045-a36dd5a7d49f" (UID: "2d474521-6484-42c0-a045-a36dd5a7d49f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:41:53.399745 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.399685 2568 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d474521-6484-42c0-a045-a36dd5a7d49f-kube-api-access-7j2tg" (OuterVolumeSpecName: "kube-api-access-7j2tg") pod "2d474521-6484-42c0-a045-a36dd5a7d49f" (UID: "2d474521-6484-42c0-a045-a36dd5a7d49f"). InnerVolumeSpecName "kube-api-access-7j2tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:41:53.498501 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.498466 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-envoy\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:41:53.498501 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.498493 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-data\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:41:53.498501 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.498503 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-workload-socket\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:41:53.498727 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.498512 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j2tg\" (UniqueName: \"kubernetes.io/projected/2d474521-6484-42c0-a045-a36dd5a7d49f-kube-api-access-7j2tg\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:41:53.498727 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.498521 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-token\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:41:53.498727 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.498529 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d474521-6484-42c0-a045-a36dd5a7d49f-credential-socket\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:41:53.498727 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:53.498541 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d474521-6484-42c0-a045-a36dd5a7d49f-istio-podinfo\") on node \"ip-10-0-141-109.ec2.internal\" DevicePath \"\"" Jul 07 19:41:54.205228 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:54.205200 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" Jul 07 19:41:54.205660 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:54.205242 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf" event={"ID":"2d474521-6484-42c0-a045-a36dd5a7d49f","Type":"ContainerDied","Data":"96cdcc8396622d80ad69c06e51cb4bc00862d0309c5868a8f6eb9402ddf821f2"} Jul 07 19:41:54.205660 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:54.205305 2568 scope.go:117] "RemoveContainer" containerID="fb89cafc032784d9f311769aa81e67479fd587ccb0fb10fd66fc51913158ebb8" Jul 07 19:41:54.234569 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:54.234534 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf"] Jul 07 19:41:54.239872 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:54.239845 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-0e4903-openshift-default-846d549df5-27dqf"] Jul 07 19:41:54.328888 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:54.328847 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d474521-6484-42c0-a045-a36dd5a7d49f" path="/var/lib/kubelet/pods/2d474521-6484-42c0-a045-a36dd5a7d49f/volumes" Jul 07 19:41:57.638617 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:41:57.638569 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c6b94b488-qmmng_8f977d89-43c2-4cfa-81d2-51879ffe4992/authorino/0.log" Jul 07 19:42:03.186469 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:03.186424 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9kfwv_184e6a6c-0a82-4c5e-957d-3a344adb0d5a/manager/0.log" Jul 07 19:42:03.578109 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:03.577994 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cc9bdf65-wnf94_fcc12345-2b14-4c96-b436-028d1a970815/copy-manifests/0.log" Jul 07 19:42:03.586607 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:03.586581 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cc9bdf65-wnf94_fcc12345-2b14-4c96-b436-028d1a970815/manager/0.log" Jul 07 19:42:04.481719 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:04.481681 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6_6a2cbdd2-093d-495d-af87-de9ea1ec7e45/util/0.log" Jul 07 19:42:04.492888 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:04.492850 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6_6a2cbdd2-093d-495d-af87-de9ea1ec7e45/pull/0.log" Jul 07 19:42:04.503416 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:04.503394 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6_6a2cbdd2-093d-495d-af87-de9ea1ec7e45/extract/0.log" Jul 07 19:42:04.620582 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:04.620551 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w_6c8b8b63-d0c6-437a-b0db-525553f775d0/pull/0.log" Jul 07 19:42:04.629313 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:04.629284 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w_6c8b8b63-d0c6-437a-b0db-525553f775d0/extract/0.log" Jul 07 19:42:04.636994 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:04.636969 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w_6c8b8b63-d0c6-437a-b0db-525553f775d0/util/0.log" Jul 07 19:42:04.750584 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:04.750484 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6_8b1b73ff-28a6-4e52-85e0-df7d72f95cb7/util/0.log" Jul 07 19:42:04.759221 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:04.759196 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6_8b1b73ff-28a6-4e52-85e0-df7d72f95cb7/pull/0.log" Jul 07 19:42:04.767710 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:04.767685 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6_8b1b73ff-28a6-4e52-85e0-df7d72f95cb7/extract/0.log" Jul 07 19:42:04.877446 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:04.877414 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd_384eb23f-f99e-4d93-b0f2-47ba2b356d99/util/0.log" Jul 07 19:42:04.885700 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:04.885669 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd_384eb23f-f99e-4d93-b0f2-47ba2b356d99/pull/0.log" Jul 07 19:42:04.894946 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:04.894925 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd_384eb23f-f99e-4d93-b0f2-47ba2b356d99/extract/0.log" Jul 07 19:42:05.013752 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:05.013653 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c6b94b488-qmmng_8f977d89-43c2-4cfa-81d2-51879ffe4992/authorino/0.log" Jul 07 19:42:05.404560 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:05.404535 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-wv44m_fb309a1e-80c0-4399-861e-2a689c8ad613/kuadrant-console-plugin/0.log" Jul 07 19:42:05.792887 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:05.792798 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-vmrzr_cbe5dc1c-b246-496e-b9ad-52598990970b/limitador/0.log" Jul 07 19:42:06.655354 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:06.655326 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cd4df5c96-dnlrc_0b034d9e-3007-4ede-86ee-3478190d159e/kube-auth-proxy/0.log" Jul 07 19:42:12.112598 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.112560 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bhqz8/must-gather-sbgl7"] Jul 07 19:42:12.113089 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.112919 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a225eb5-10f7-4638-86f6-fb0d7208c70c" containerName="main" Jul 07 19:42:12.113089 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.112932 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a225eb5-10f7-4638-86f6-fb0d7208c70c" containerName="main" Jul 07 19:42:12.113089 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.112983 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d474521-6484-42c0-a045-a36dd5a7d49f" containerName="istio-proxy" Jul 07 19:42:12.113089 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.112990 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d474521-6484-42c0-a045-a36dd5a7d49f" containerName="istio-proxy" Jul 07 19:42:12.113089 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.113002 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d474521-6484-42c0-a045-a36dd5a7d49f" containerName="istio-proxy" Jul 07 19:42:12.113089 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.113021 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d474521-6484-42c0-a045-a36dd5a7d49f" containerName="istio-proxy" Jul 07 19:42:12.113502 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.113137 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d474521-6484-42c0-a045-a36dd5a7d49f" containerName="istio-proxy" Jul 07 19:42:12.113502 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.113152 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d474521-6484-42c0-a045-a36dd5a7d49f" containerName="istio-proxy" Jul 07 19:42:12.113502 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.113165 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a225eb5-10f7-4638-86f6-fb0d7208c70c" containerName="main" Jul 07 19:42:12.116524 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.116503 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bhqz8/must-gather-sbgl7" Jul 07 19:42:12.121735 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.121712 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bhqz8\"/\"default-dockercfg-b4nc4\"" Jul 07 19:42:12.122484 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.122461 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bhqz8\"/\"kube-root-ca.crt\"" Jul 07 19:42:12.122565 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.122465 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bhqz8\"/\"openshift-service-ca.crt\"" Jul 07 19:42:12.129892 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.129865 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bhqz8/must-gather-sbgl7"] Jul 07 19:42:12.262723 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.262679 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e49f4666-edb9-42a8-bd4b-7d173c3dab52-must-gather-output\") pod \"must-gather-sbgl7\" (UID: \"e49f4666-edb9-42a8-bd4b-7d173c3dab52\") " pod="openshift-must-gather-bhqz8/must-gather-sbgl7" Jul 07 19:42:12.262907 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.262749 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chc42\" (UniqueName: \"kubernetes.io/projected/e49f4666-edb9-42a8-bd4b-7d173c3dab52-kube-api-access-chc42\") pod \"must-gather-sbgl7\" (UID: \"e49f4666-edb9-42a8-bd4b-7d173c3dab52\") " pod="openshift-must-gather-bhqz8/must-gather-sbgl7" Jul 07 19:42:12.363458 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.363357 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e49f4666-edb9-42a8-bd4b-7d173c3dab52-must-gather-output\") pod \"must-gather-sbgl7\" (UID: \"e49f4666-edb9-42a8-bd4b-7d173c3dab52\") " pod="openshift-must-gather-bhqz8/must-gather-sbgl7" Jul 07 19:42:12.363458 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.363418 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chc42\" (UniqueName: \"kubernetes.io/projected/e49f4666-edb9-42a8-bd4b-7d173c3dab52-kube-api-access-chc42\") pod \"must-gather-sbgl7\" (UID: \"e49f4666-edb9-42a8-bd4b-7d173c3dab52\") " pod="openshift-must-gather-bhqz8/must-gather-sbgl7" Jul 07 19:42:12.363704 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.363684 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e49f4666-edb9-42a8-bd4b-7d173c3dab52-must-gather-output\") pod \"must-gather-sbgl7\" (UID: \"e49f4666-edb9-42a8-bd4b-7d173c3dab52\") " pod="openshift-must-gather-bhqz8/must-gather-sbgl7" Jul 07 19:42:12.373863 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.373834 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chc42\" (UniqueName: \"kubernetes.io/projected/e49f4666-edb9-42a8-bd4b-7d173c3dab52-kube-api-access-chc42\") pod \"must-gather-sbgl7\" (UID: \"e49f4666-edb9-42a8-bd4b-7d173c3dab52\") " pod="openshift-must-gather-bhqz8/must-gather-sbgl7" Jul 07 19:42:12.425327 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.425288 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bhqz8/must-gather-sbgl7" Jul 07 19:42:12.565513 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.565484 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bhqz8/must-gather-sbgl7"] Jul 07 19:42:12.567794 ip-10-0-141-109 kubenswrapper[2568]: W0707 19:42:12.567764 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode49f4666_edb9_42a8_bd4b_7d173c3dab52.slice/crio-92e2831cba10f09567bb84ad39206d2ebab23debcd31b213fd2ebe70df28dccc WatchSource:0}: Error finding container 92e2831cba10f09567bb84ad39206d2ebab23debcd31b213fd2ebe70df28dccc: Status 404 returned error can't find the container with id 92e2831cba10f09567bb84ad39206d2ebab23debcd31b213fd2ebe70df28dccc Jul 07 19:42:12.569633 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:12.569617 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:42:13.304991 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:13.304953 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bhqz8/must-gather-sbgl7" event={"ID":"e49f4666-edb9-42a8-bd4b-7d173c3dab52","Type":"ContainerStarted","Data":"92e2831cba10f09567bb84ad39206d2ebab23debcd31b213fd2ebe70df28dccc"} Jul 07 19:42:14.312900 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:14.312859 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bhqz8/must-gather-sbgl7" event={"ID":"e49f4666-edb9-42a8-bd4b-7d173c3dab52","Type":"ContainerStarted","Data":"ac1c9d1c29072a699856e40ec3fcf314362bce6c415342f962133c4c13a2afe5"} Jul 07 19:42:15.320234 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:15.320193 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bhqz8/must-gather-sbgl7" event={"ID":"e49f4666-edb9-42a8-bd4b-7d173c3dab52","Type":"ContainerStarted","Data":"22f37260b45842803100ff24f79fd54cd4a1997f2387468cc5975c54f291a7aa"} Jul 07 19:42:15.345339 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:15.345276 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bhqz8/must-gather-sbgl7" podStartSLOduration=1.80750695 podStartE2EDuration="3.345253505s" totalImagesPullingTime="1.537746555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:42:15.343195129 +0000 UTC m=+2505.535735201" watchObservedRunningTime="2026-07-07 19:42:15.345253505 +0000 UTC m=+2505.537793574" Jul 07 19:42:16.090289 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:16.090260 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n6cxp_c14f74dc-d3d9-4962-9dbc-7a3209b3d4ac/global-pull-secret-syncer/0.log" Jul 07 19:42:16.220899 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:16.220848 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hvj8p_fa2994d8-22f9-4a87-92c2-71f47f44fd4e/konnectivity-agent/0.log" Jul 07 19:42:16.348630 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:16.348552 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-109.ec2.internal_3bad345f280d0902b6e7e2ae30ad5b4d/haproxy/0.log" Jul 07 19:42:20.498936 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:20.498898 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6_6a2cbdd2-093d-495d-af87-de9ea1ec7e45/extract/0.log" Jul 07 19:42:20.561361 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:20.561333 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6_6a2cbdd2-093d-495d-af87-de9ea1ec7e45/util/0.log" Jul 07 19:42:20.613451 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:20.613417 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759599b6_6a2cbdd2-093d-495d-af87-de9ea1ec7e45/pull/0.log" Jul 07 19:42:20.687807 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:20.687773 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w_6c8b8b63-d0c6-437a-b0db-525553f775d0/extract/0.log" Jul 07 19:42:20.734211 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:20.734174 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w_6c8b8b63-d0c6-437a-b0db-525553f775d0/util/0.log" Jul 07 19:42:20.769685 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:20.769600 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xh86w_6c8b8b63-d0c6-437a-b0db-525553f775d0/pull/0.log" Jul 07 19:42:20.822350 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:20.822313 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6_8b1b73ff-28a6-4e52-85e0-df7d72f95cb7/extract/0.log" Jul 07 19:42:20.868512 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:20.868470 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6_8b1b73ff-28a6-4e52-85e0-df7d72f95cb7/util/0.log" Jul 07 19:42:20.906418 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:20.906376 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xcdt6_8b1b73ff-28a6-4e52-85e0-df7d72f95cb7/pull/0.log" Jul 07 19:42:20.962680 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:20.962651 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd_384eb23f-f99e-4d93-b0f2-47ba2b356d99/extract/0.log" Jul 07 19:42:20.994892 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:20.994865 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd_384eb23f-f99e-4d93-b0f2-47ba2b356d99/util/0.log" Jul 07 19:42:21.047289 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:21.047194 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jr5xd_384eb23f-f99e-4d93-b0f2-47ba2b356d99/pull/0.log" Jul 07 19:42:21.393161 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:21.393119 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c6b94b488-qmmng_8f977d89-43c2-4cfa-81d2-51879ffe4992/authorino/0.log" Jul 07 19:42:21.666810 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:21.666735 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-wv44m_fb309a1e-80c0-4399-861e-2a689c8ad613/kuadrant-console-plugin/0.log" Jul 07 19:42:22.065150 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:22.064993 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-vmrzr_cbe5dc1c-b246-496e-b9ad-52598990970b/limitador/0.log" Jul 07 19:42:24.487874 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:24.487834 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-cdfrm_db8101fa-7197-464d-b9a8-385f5629f7ff/kube-state-metrics/0.log" Jul 07 19:42:24.525260 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:24.525222 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-cdfrm_db8101fa-7197-464d-b9a8-385f5629f7ff/kube-rbac-proxy-main/0.log" Jul 07 19:42:24.561226 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:24.561189 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-cdfrm_db8101fa-7197-464d-b9a8-385f5629f7ff/kube-rbac-proxy-self/0.log" Jul 07 19:42:24.660669 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:24.660634 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-k98wx_890a28fe-4573-40f6-8336-eabf828bab94/monitoring-plugin/0.log" Jul 07 19:42:24.897017 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:24.896989 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rvlhp_ff8991a0-e43b-429f-bba8-6afbd5dd08ee/node-exporter/0.log" Jul 07 19:42:24.928947 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:24.928910 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rvlhp_ff8991a0-e43b-429f-bba8-6afbd5dd08ee/kube-rbac-proxy/0.log" Jul 07 19:42:24.961277 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:24.961251 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rvlhp_ff8991a0-e43b-429f-bba8-6afbd5dd08ee/init-textfile/0.log" Jul 07 19:42:25.146400 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.146364 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-222dv_c0f243a4-a647-4eb4-bdbf-16638f205522/kube-rbac-proxy-main/0.log" Jul 07 19:42:25.184334 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.184250 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-222dv_c0f243a4-a647-4eb4-bdbf-16638f205522/kube-rbac-proxy-self/0.log" Jul 07 19:42:25.214491 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.214448 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-222dv_c0f243a4-a647-4eb4-bdbf-16638f205522/openshift-state-metrics/0.log" Jul 07 19:42:25.618748 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.618658 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2"] Jul 07 19:42:25.624671 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.624644 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" Jul 07 19:42:25.638679 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.638649 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2"] Jul 07 19:42:25.748427 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.748398 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-d9lht_2bd2a75e-64e6-4544-b6ce-b705f34214f3/prometheus-operator-admission-webhook/0.log" Jul 07 19:42:25.802106 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.802073 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f6401e4-96f0-43ec-beed-1f723287ba73-lib-modules\") pod \"perf-node-gather-daemonset-d79w2\" (UID: \"4f6401e4-96f0-43ec-beed-1f723287ba73\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" Jul 07 19:42:25.802282 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.802120 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f6401e4-96f0-43ec-beed-1f723287ba73-podres\") pod \"perf-node-gather-daemonset-d79w2\" (UID: \"4f6401e4-96f0-43ec-beed-1f723287ba73\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" Jul 07 19:42:25.802282 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.802159 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f6401e4-96f0-43ec-beed-1f723287ba73-sys\") pod \"perf-node-gather-daemonset-d79w2\" (UID: \"4f6401e4-96f0-43ec-beed-1f723287ba73\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" Jul 07 19:42:25.802282 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.802197 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtprk\" (UniqueName: \"kubernetes.io/projected/4f6401e4-96f0-43ec-beed-1f723287ba73-kube-api-access-xtprk\") pod \"perf-node-gather-daemonset-d79w2\" (UID: \"4f6401e4-96f0-43ec-beed-1f723287ba73\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" Jul 07 19:42:25.802282 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.802244 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f6401e4-96f0-43ec-beed-1f723287ba73-proc\") pod \"perf-node-gather-daemonset-d79w2\" (UID: \"4f6401e4-96f0-43ec-beed-1f723287ba73\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" Jul 07 19:42:25.892299 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.892265 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69b8b8d8d4-qfsbr_213c5b68-851c-48c2-922b-120d4c007aed/thanos-query/0.log" Jul 07 19:42:25.903601 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.903556 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f6401e4-96f0-43ec-beed-1f723287ba73-lib-modules\") pod \"perf-node-gather-daemonset-d79w2\" (UID: \"4f6401e4-96f0-43ec-beed-1f723287ba73\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" Jul 07 19:42:25.903601 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.903606 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f6401e4-96f0-43ec-beed-1f723287ba73-podres\") pod \"perf-node-gather-daemonset-d79w2\" (UID: \"4f6401e4-96f0-43ec-beed-1f723287ba73\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" Jul 07 19:42:25.903857 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.903644 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f6401e4-96f0-43ec-beed-1f723287ba73-sys\") pod \"perf-node-gather-daemonset-d79w2\" (UID: \"4f6401e4-96f0-43ec-beed-1f723287ba73\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" Jul 07 19:42:25.903857 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.903687 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtprk\" (UniqueName: \"kubernetes.io/projected/4f6401e4-96f0-43ec-beed-1f723287ba73-kube-api-access-xtprk\") pod \"perf-node-gather-daemonset-d79w2\" (UID: \"4f6401e4-96f0-43ec-beed-1f723287ba73\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" Jul 07 19:42:25.903857 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.903728 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f6401e4-96f0-43ec-beed-1f723287ba73-proc\") pod \"perf-node-gather-daemonset-d79w2\" (UID: \"4f6401e4-96f0-43ec-beed-1f723287ba73\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" Jul 07 19:42:25.903857 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.903764 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f6401e4-96f0-43ec-beed-1f723287ba73-lib-modules\") pod \"perf-node-gather-daemonset-d79w2\" (UID: \"4f6401e4-96f0-43ec-beed-1f723287ba73\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" Jul 07 19:42:25.903857 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.903788 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f6401e4-96f0-43ec-beed-1f723287ba73-sys\") pod \"perf-node-gather-daemonset-d79w2\" (UID: \"4f6401e4-96f0-43ec-beed-1f723287ba73\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" Jul 07 19:42:25.903857 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.903816 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4f6401e4-96f0-43ec-beed-1f723287ba73-podres\") pod \"perf-node-gather-daemonset-d79w2\" (UID: \"4f6401e4-96f0-43ec-beed-1f723287ba73\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" Jul 07 19:42:25.903857 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.903825 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4f6401e4-96f0-43ec-beed-1f723287ba73-proc\") pod \"perf-node-gather-daemonset-d79w2\" (UID: \"4f6401e4-96f0-43ec-beed-1f723287ba73\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" Jul 07 19:42:25.915295 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.915218 2568 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtprk\" (UniqueName: \"kubernetes.io/projected/4f6401e4-96f0-43ec-beed-1f723287ba73-kube-api-access-xtprk\") pod \"perf-node-gather-daemonset-d79w2\" (UID: \"4f6401e4-96f0-43ec-beed-1f723287ba73\") " pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" Jul 07 19:42:25.924431 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.924277 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69b8b8d8d4-qfsbr_213c5b68-851c-48c2-922b-120d4c007aed/kube-rbac-proxy-web/0.log" Jul 07 19:42:25.938867 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.938826 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" Jul 07 19:42:25.955687 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.955655 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69b8b8d8d4-qfsbr_213c5b68-851c-48c2-922b-120d4c007aed/kube-rbac-proxy/0.log" Jul 07 19:42:25.987777 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:25.987743 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69b8b8d8d4-qfsbr_213c5b68-851c-48c2-922b-120d4c007aed/prom-label-proxy/0.log" Jul 07 19:42:26.019526 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:26.019464 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69b8b8d8d4-qfsbr_213c5b68-851c-48c2-922b-120d4c007aed/kube-rbac-proxy-rules/0.log" Jul 07 19:42:26.063298 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:26.063243 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69b8b8d8d4-qfsbr_213c5b68-851c-48c2-922b-120d4c007aed/kube-rbac-proxy-metrics/0.log" Jul 07 19:42:26.121918 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:26.121685 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2"] Jul 07 19:42:26.415698 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:26.415612 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" event={"ID":"4f6401e4-96f0-43ec-beed-1f723287ba73","Type":"ContainerStarted","Data":"054b32eea0f1496c2fa8a45f713dc11ba7dfc6a7bbc5975188e98341c9632081"} Jul 07 19:42:26.415698 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:26.415678 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" event={"ID":"4f6401e4-96f0-43ec-beed-1f723287ba73","Type":"ContainerStarted","Data":"b122517c08679cf259a6d752ed60f68da9ee3ad355b5757bd452b10e9b96766f"} Jul 07 19:42:26.416028 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:26.415722 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" Jul 07 19:42:26.442078 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:26.441974 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" podStartSLOduration=1.441951277 podStartE2EDuration="1.441951277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:42:26.438239087 +0000 UTC m=+2516.630779153" watchObservedRunningTime="2026-07-07 19:42:26.441951277 +0000 UTC m=+2516.634491345" Jul 07 19:42:27.235527 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:27.235496 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-6487b_cadb609a-f2e9-4ca0-a4eb-85952ff8160c/networking-console-plugin/0.log" Jul 07 19:42:28.452155 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:28.452123 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-58d6f4c969-pm94w_bc80c657-dc92-43e9-9f27-89c656321e52/console/0.log" Jul 07 19:42:28.491210 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:28.491175 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-9cbxm_6ab43ed2-15f4-4bbd-ab1d-a06533c1afd7/download-server/0.log" Jul 07 19:42:29.171011 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:29.170980 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-rg2g5_88388786-b65f-4506-a3c5-808df99f9fa7/volume-data-source-validator/0.log" Jul 07 19:42:30.432139 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:30.432108 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hrh48_34b59a25-fd9d-4ed1-befe-7a9d4ad5db74/dns/0.log" Jul 07 19:42:30.460921 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:30.460892 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hrh48_34b59a25-fd9d-4ed1-befe-7a9d4ad5db74/kube-rbac-proxy/0.log" Jul 07 19:42:30.600784 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:30.600760 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ptk5w_a079670f-9732-4279-901b-5fc3310a9a84/dns-node-resolver/0.log" Jul 07 19:42:31.260348 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:31.260318 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-76f5b666df-rl2xc_9a2baf66-4d94-4bca-b8c8-7d64df15ad0e/registry/0.log" Jul 07 19:42:31.384308 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:31.384281 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vrwmc_61a2f3ec-3a02-49c4-984b-ca722d4f7914/node-ca/0.log" Jul 07 19:42:32.434429 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:32.434387 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bhqz8/perf-node-gather-daemonset-d79w2" Jul 07 19:42:32.745487 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:32.745382 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cd4df5c96-dnlrc_0b034d9e-3007-4ede-86ee-3478190d159e/kube-auth-proxy/0.log" Jul 07 19:42:33.952871 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:33.952812 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vz9jl_08cae73a-9871-4cf5-a4d5-1045cc284553/serve-healthcheck-canary/0.log" Jul 07 19:42:34.653638 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:34.653608 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-j7h8x_32659884-db20-426c-be99-476c7718c18d/insights-operator/0.log" Jul 07 19:42:34.654334 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:34.654315 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-j7h8x_32659884-db20-426c-be99-476c7718c18d/insights-operator/1.log" Jul 07 19:42:35.022148 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:35.022112 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-trklp_d200611b-531a-49e6-8d44-20ba3ac01a4a/kube-rbac-proxy/0.log" Jul 07 19:42:35.060792 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:35.060757 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-trklp_d200611b-531a-49e6-8d44-20ba3ac01a4a/exporter/0.log" Jul 07 19:42:35.100690 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:35.100660 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-trklp_d200611b-531a-49e6-8d44-20ba3ac01a4a/extractor/0.log" Jul 07 19:42:36.577194 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:36.577161 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cd4df5c96-dnlrc_0b034d9e-3007-4ede-86ee-3478190d159e/kube-auth-proxy/0.log" Jul 07 19:42:38.328532 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:38.328500 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cd4df5c96-dnlrc_0b034d9e-3007-4ede-86ee-3478190d159e/kube-auth-proxy/0.log" Jul 07 19:42:38.441339 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:38.441305 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9kfwv_184e6a6c-0a82-4c5e-957d-3a344adb0d5a/manager/0.log" Jul 07 19:42:38.901336 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:38.901304 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cc9bdf65-wnf94_fcc12345-2b14-4c96-b436-028d1a970815/manager/0.log" Jul 07 19:42:38.950350 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:38.950319 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cc9bdf65-wnf94_fcc12345-2b14-4c96-b436-028d1a970815/copy-manifests/0.log" Jul 07 19:42:40.489298 ip-10-0-141-109 kubenswrapper[2568]: I0707 19:42:40.489251 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5f4bfc584b-mfmml_6c0c2e5b-c75b-4122-9eb8-1db975b14e53/manager/0.log"