Jul 24 15:43:15.695593 ip-10-0-141-58 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 24 15:43:15.695602 ip-10-0-141-58 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 24 15:43:15.695609 ip-10-0-141-58 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 24 15:43:15.695873 ip-10-0-141-58 systemd[1]: Failed to start Kubernetes Kubelet. Jul 24 15:43:25.871724 ip-10-0-141-58 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 24 15:43:25.871744 ip-10-0-141-58 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot adbfbf098d524e68aa4c49759a987a94 -- Jul 24 15:45:23.692786 ip-10-0-141-58 systemd[1]: Starting Kubernetes Kubelet... Jul 24 15:45:24.044928 ip-10-0-141-58 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 15:45:24.044928 ip-10-0-141-58 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 15:45:24.044928 ip-10-0-141-58 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 15:45:24.044928 ip-10-0-141-58 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 15:45:24.044928 ip-10-0-141-58 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 15:45:24.045668 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.044979 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 15:45:24.050723 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050703 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:45:24.050723 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050723 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:45:24.050723 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050728 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:45:24.050837 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050731 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:45:24.050837 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050734 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:45:24.050837 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050738 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:45:24.050837 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050740 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:45:24.050837 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050744 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:45:24.050837 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050747 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:45:24.050837 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050750 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:45:24.050837 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050753 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:45:24.050837 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050755 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:45:24.050837 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050763 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:45:24.050837 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050767 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:45:24.050837 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050770 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:45:24.050837 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050773 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:45:24.050837 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050776 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:45:24.050837 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050780 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:45:24.050837 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050784 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:45:24.050837 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050787 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:45:24.050837 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050790 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:45:24.050837 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050793 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:45:24.051333 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050796 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:45:24.051333 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050798 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:45:24.051333 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050801 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:45:24.051333 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050805 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:45:24.051333 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050809 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:45:24.051333 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050812 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:45:24.051333 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050815 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:45:24.051333 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050818 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:45:24.051333 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050821 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:45:24.051333 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050823 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:45:24.051333 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050826 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:45:24.051333 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050828 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:45:24.051333 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050831 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:45:24.051333 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050834 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:45:24.051333 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050837 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:45:24.051333 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050840 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:45:24.051333 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050843 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:45:24.051333 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050846 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:45:24.051333 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050849 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050852 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050854 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050857 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050860 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050862 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050865 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050867 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050870 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050873 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050876 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050878 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050881 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050883 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050886 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050888 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050891 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050894 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050896 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050899 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:45:24.051815 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050901 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050904 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050906 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050909 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050911 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050914 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050919 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050922 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050924 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050927 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050930 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050933 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050936 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050940 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050943 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050946 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050949 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050952 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050955 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050958 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:45:24.052310 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050960 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:45:24.052914 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050963 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:45:24.052914 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050966 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:45:24.052914 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050969 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:45:24.052914 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050971 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:45:24.052914 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050974 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:45:24.052914 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050977 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:45:24.052914 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050980 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:45:24.052914 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050982 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:45:24.052914 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050985 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:45:24.052914 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050987 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:45:24.052914 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050990 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:45:24.052914 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050993 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:45:24.052914 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050995 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:45:24.052914 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.050998 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:45:24.052914 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051000 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:45:24.052914 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051003 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:45:24.052914 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051005 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:45:24.052914 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051008 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:45:24.052914 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051742 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051750 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051753 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051756 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051758 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051762 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051764 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051767 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051770 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051773 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051775 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051778 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051780 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051783 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051785 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051789 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051793 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051797 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051800 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051803 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:45:24.053385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051805 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:45:24.053905 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051808 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:45:24.053905 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051812 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:45:24.053905 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051816 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:45:24.053905 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051819 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:45:24.053905 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051822 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:45:24.053905 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051825 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:45:24.053905 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051828 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:45:24.053905 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051830 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:45:24.053905 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051833 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:45:24.053905 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051835 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:45:24.053905 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051838 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:45:24.053905 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051841 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:45:24.053905 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051844 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:45:24.053905 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051847 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:45:24.053905 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051850 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:45:24.053905 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051852 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:45:24.053905 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051855 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:45:24.053905 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051858 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:45:24.053905 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051861 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051864 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051867 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051869 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051873 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051875 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051878 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051880 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051883 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051885 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051888 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051890 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051893 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051895 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051898 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051900 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051903 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051905 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051908 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051911 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:45:24.054376 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051913 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:45:24.054904 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051917 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:45:24.054904 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051919 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:45:24.054904 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051922 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:45:24.054904 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051924 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:45:24.054904 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051933 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:45:24.054904 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051937 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:45:24.054904 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051941 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:45:24.054904 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051943 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:45:24.054904 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051946 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:45:24.054904 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051948 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:45:24.054904 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051951 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:45:24.054904 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051955 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:45:24.054904 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051957 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:45:24.054904 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051960 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:45:24.054904 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051962 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:45:24.054904 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051965 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:45:24.054904 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051968 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:45:24.054904 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051970 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:45:24.054904 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051973 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051975 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051978 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051981 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051983 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051986 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051988 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051991 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051994 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.051997 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052000 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052002 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052005 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052007 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052010 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052012 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052015 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052017 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052021 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052024 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:45:24.055404 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052026 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052410 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052416 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052419 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052421 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052425 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052428 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052431 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052433 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052436 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052438 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052441 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052443 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052446 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052463 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052466 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052469 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052477 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052481 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052484 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:45:24.055906 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052487 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052491 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052494 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052496 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052499 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052502 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052504 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052507 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052509 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052512 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052517 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052519 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052522 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052524 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052527 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052529 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052532 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052535 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052539 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052542 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:45:24.056396 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052544 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:45:24.056946 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052547 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:45:24.056946 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052549 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:45:24.056946 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052552 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:45:24.056946 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052554 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:45:24.056946 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052557 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:45:24.056946 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052559 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:45:24.056946 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052562 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:45:24.056946 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052564 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:45:24.056946 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052566 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:45:24.056946 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052569 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:45:24.056946 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052572 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:45:24.056946 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052574 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:45:24.056946 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052577 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:45:24.056946 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052580 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:45:24.056946 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052583 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:45:24.056946 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052585 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:45:24.056946 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052589 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:45:24.056946 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052593 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:45:24.056946 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052597 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:45:24.057412 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052600 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:45:24.057412 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052602 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:45:24.057412 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052607 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:45:24.057412 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052610 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:45:24.057412 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052612 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:45:24.057412 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052615 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:45:24.057412 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052618 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:45:24.057412 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052621 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:45:24.057412 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052624 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:45:24.057412 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052627 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:45:24.057412 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052630 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:45:24.057412 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052633 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:45:24.057412 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052636 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:45:24.057412 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052640 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:45:24.057412 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052642 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:45:24.057412 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052645 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:45:24.057412 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052648 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:45:24.057412 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052651 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:45:24.057412 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052653 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:45:24.057956 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052656 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:45:24.057956 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052660 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:45:24.057956 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052663 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:45:24.057956 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052667 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:45:24.057956 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052670 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:45:24.057956 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052672 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:45:24.057956 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052676 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:45:24.057956 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052679 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:45:24.057956 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052682 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:45:24.057956 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052684 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:45:24.057956 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052687 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:45:24.057956 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052690 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:45:24.057956 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052692 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:45:24.057956 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052695 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:45:24.057956 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052697 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:45:24.057956 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052701 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:45:24.057956 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052704 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:45:24.057956 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052706 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:45:24.057956 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052709 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052711 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.052714 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052793 2569 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052801 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052808 2569 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052813 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052818 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052822 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052827 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052831 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052835 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052838 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052842 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052845 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052848 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052851 2569 flags.go:64] FLAG: --cgroup-root="" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052854 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052857 2569 flags.go:64] FLAG: --client-ca-file="" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052860 2569 flags.go:64] FLAG: --cloud-provider="external" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052863 2569 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052868 2569 flags.go:64] FLAG: --cluster-domain="" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052871 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 15:45:24.058465 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052874 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052877 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052881 2569 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052885 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052888 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052891 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052896 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052899 2569 flags.go:64] FLAG: --contention-profiling="false" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052902 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052905 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052908 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052911 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052916 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052919 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052923 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052926 2569 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052929 2569 flags.go:64] FLAG: --enable-server="true" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052933 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052937 2569 flags.go:64] FLAG: --event-burst="100" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052940 2569 flags.go:64] FLAG: --event-qps="50" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052943 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052946 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052949 2569 flags.go:64] FLAG: --eviction-hard="" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052953 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 15:45:24.059027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052956 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052959 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052962 2569 flags.go:64] FLAG: --eviction-soft="" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052964 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052970 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052973 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052976 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052979 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052982 2569 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052985 2569 flags.go:64] FLAG: --feature-gates="" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052989 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052992 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052996 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.052999 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053003 2569 flags.go:64] FLAG: --healthz-port="10248" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053006 2569 flags.go:64] FLAG: --help="false" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053009 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-141-58.ec2.internal" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053012 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053015 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053018 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053022 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053025 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053028 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 15:45:24.059634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053031 2569 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053033 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053036 2569 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053039 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053043 2569 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053046 2569 flags.go:64] FLAG: --kube-reserved="" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053049 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053052 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053055 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053058 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053061 2569 flags.go:64] FLAG: --lock-file="" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053064 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053067 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053072 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053077 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053080 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053083 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053086 2569 flags.go:64] FLAG: --logging-format="text" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053089 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053092 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053095 2569 flags.go:64] FLAG: --manifest-url="" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053098 2569 flags.go:64] FLAG: --manifest-url-header="" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053103 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053106 2569 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053117 2569 flags.go:64] FLAG: --max-pods="110" Jul 24 15:45:24.060221 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053120 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053123 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053126 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053129 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053131 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053135 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053137 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053145 2569 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053148 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053151 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053154 2569 flags.go:64] FLAG: --pod-cidr="" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053158 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053163 2569 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053166 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053170 2569 flags.go:64] FLAG: --pods-per-core="0" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053172 2569 flags.go:64] FLAG: --port="10250" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053175 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053178 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07ee56984d9d83ab2" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053182 2569 flags.go:64] FLAG: --qos-reserved="" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053185 2569 flags.go:64] FLAG: --read-only-port="10255" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053188 2569 flags.go:64] FLAG: --register-node="true" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053192 2569 flags.go:64] FLAG: --register-with-taints="" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053196 2569 flags.go:64] FLAG: --registry-burst="10" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053198 2569 flags.go:64] FLAG: --registry-qps="5" Jul 24 15:45:24.060973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053201 2569 flags.go:64] FLAG: --reserved-cpus="" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053204 2569 flags.go:64] FLAG: --reserved-memory="" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053208 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053211 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053214 2569 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053217 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053220 2569 flags.go:64] FLAG: --runonce="false" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053223 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053226 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053229 2569 flags.go:64] FLAG: --seccomp-default="false" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053232 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053234 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053238 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053241 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053244 2569 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053247 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053250 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053252 2569 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053255 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053259 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053262 2569 flags.go:64] FLAG: --system-cgroups="" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053265 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053271 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053274 2569 flags.go:64] FLAG: --tls-cert-file="" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053277 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 15:45:24.061616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053282 2569 flags.go:64] FLAG: --tls-min-version="" Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053285 2569 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053288 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053291 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053295 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053298 2569 flags.go:64] FLAG: --v="2" Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053302 2569 flags.go:64] FLAG: --version="false" Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053307 2569 flags.go:64] FLAG: --vmodule="" Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053310 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.053313 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053471 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053476 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053480 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053484 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053488 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053490 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053493 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053496 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053499 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053502 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053504 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053507 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:45:24.062255 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053510 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:45:24.062819 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053512 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:45:24.062819 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053515 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:45:24.062819 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053518 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:45:24.062819 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053520 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:45:24.062819 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053523 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:45:24.062819 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053526 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:45:24.062819 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053529 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:45:24.062819 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053531 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:45:24.062819 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053534 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:45:24.062819 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053537 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:45:24.062819 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053539 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:45:24.062819 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053542 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:45:24.062819 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053544 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:45:24.062819 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053548 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:45:24.062819 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053551 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:45:24.062819 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053554 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:45:24.062819 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053556 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:45:24.062819 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053559 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:45:24.062819 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053561 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053564 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053566 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053571 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053574 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053576 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053579 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053581 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053584 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053587 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053590 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053593 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053595 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053598 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053601 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053604 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053606 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053609 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053612 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053615 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:45:24.063290 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053618 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:45:24.063891 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053620 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:45:24.063891 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053623 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:45:24.063891 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053626 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:45:24.063891 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053628 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:45:24.063891 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053632 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:45:24.063891 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053635 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:45:24.063891 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053639 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:45:24.063891 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053642 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:45:24.063891 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053645 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:45:24.063891 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053647 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:45:24.063891 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053650 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:45:24.063891 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053652 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:45:24.063891 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053655 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:45:24.063891 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053657 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:45:24.063891 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053662 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:45:24.063891 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053664 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:45:24.063891 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053667 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:45:24.063891 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053671 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:45:24.063891 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053675 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053678 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053680 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053683 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053686 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053688 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053691 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053693 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053696 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053699 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053701 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053704 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053706 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053709 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053715 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053718 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053720 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053723 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053726 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053728 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:45:24.064537 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053732 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:45:24.065086 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053735 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:45:24.065086 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053738 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:45:24.065086 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053741 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:45:24.065086 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053745 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:45:24.065086 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053748 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:45:24.065086 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053751 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:45:24.065086 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053754 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:45:24.065086 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.053758 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:45:24.065086 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.054516 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 15:45:24.065086 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.062222 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 15:45:24.065086 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.062241 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 15:45:24.065086 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.062256 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 15:45:24.065086 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.062263 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 15:45:24.065086 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062325 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062332 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062336 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062339 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062342 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062345 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062348 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062351 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062354 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062357 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062360 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062363 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062366 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062369 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062372 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062374 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062377 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062379 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062382 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062384 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:45:24.065448 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062388 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062390 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062393 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062395 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062399 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062401 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062404 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062406 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062409 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062414 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062416 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062419 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062422 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062425 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062428 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062430 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062434 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062436 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062439 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062442 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:45:24.065970 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062444 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:45:24.066557 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062447 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:45:24.066557 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062449 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:45:24.066557 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062469 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:45:24.066557 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062472 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:45:24.066557 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062474 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:45:24.066557 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062477 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:45:24.066557 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062480 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:45:24.066557 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062483 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:45:24.066557 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062485 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:45:24.066557 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062488 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:45:24.066557 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062491 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:45:24.066557 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062494 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:45:24.066557 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062497 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:45:24.066557 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062499 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:45:24.066557 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062502 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:45:24.066557 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062504 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:45:24.066557 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062507 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:45:24.066557 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062511 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:45:24.066557 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062515 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:45:24.067037 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062518 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:45:24.067037 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062521 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:45:24.067037 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062526 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:45:24.067037 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062529 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:45:24.067037 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062532 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:45:24.067037 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062534 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:45:24.067037 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062537 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:45:24.067037 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062540 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:45:24.067037 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062542 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:45:24.067037 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062545 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:45:24.067037 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062547 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:45:24.067037 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062550 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:45:24.067037 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062552 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:45:24.067037 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062555 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:45:24.067037 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062558 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:45:24.067037 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062560 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:45:24.067037 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062563 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:45:24.067037 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062565 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:45:24.067037 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062568 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062572 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062576 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062578 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062582 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062584 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062587 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062590 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062592 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062595 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062597 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062600 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062603 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062606 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062609 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062612 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062614 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062618 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062620 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062623 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:45:24.067527 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062625 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:45:24.068021 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.062631 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 15:45:24.068021 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062751 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:45:24.068021 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062757 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:45:24.068021 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062760 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:45:24.068021 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062763 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:45:24.068021 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062766 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:45:24.068021 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062769 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:45:24.068021 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062772 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:45:24.068021 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062774 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:45:24.068021 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062777 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:45:24.068021 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062781 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:45:24.068021 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062785 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:45:24.068021 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062788 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:45:24.068021 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062791 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:45:24.068380 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062793 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:45:24.068380 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062796 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:45:24.068380 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062799 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:45:24.068380 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062801 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:45:24.068380 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062805 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:45:24.068380 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062809 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:45:24.068380 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062812 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:45:24.068380 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062814 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:45:24.068380 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062817 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:45:24.068380 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062820 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:45:24.068380 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062823 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:45:24.068380 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062826 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:45:24.068380 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062829 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:45:24.068380 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062832 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:45:24.068380 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062835 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:45:24.068380 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062839 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:45:24.068380 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062842 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:45:24.068380 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062845 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:45:24.068380 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062847 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062850 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062853 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062856 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062858 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062861 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062863 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062866 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062868 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062871 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062873 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062876 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062878 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062880 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062883 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062886 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062888 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062891 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062895 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062899 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:45:24.068882 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062901 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062905 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062908 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062910 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062914 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062917 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062920 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062923 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062925 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062928 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062931 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062933 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062936 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062938 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062941 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062944 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062946 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062948 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062951 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062953 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:45:24.069363 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062956 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062959 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062961 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062964 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062966 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062969 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062971 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062974 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062976 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062979 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062981 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062984 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062986 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062990 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062992 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062995 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.062998 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.063000 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.063003 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.063005 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:45:24.070008 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.063008 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:45:24.070528 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.063010 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:45:24.070528 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.063013 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:45:24.070528 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.063016 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:45:24.070528 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.063019 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:45:24.070528 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.063021 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:45:24.070528 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.063024 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:45:24.070528 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:24.063027 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:45:24.070528 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.063032 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 15:45:24.070528 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.063163 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 15:45:24.070528 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.066229 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 15:45:24.070528 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.067626 2569 server.go:1019] "Starting client certificate rotation" Jul 24 15:45:24.070528 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.067724 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 15:45:24.070528 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.067761 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 15:45:24.070528 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.069969 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 15:45:24.071821 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.071807 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 15:45:24.084361 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.084340 2569 log.go:25] "Validated CRI v1 runtime API" Jul 24 15:45:24.086645 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.086630 2569 log.go:25] "Validated CRI v1 image API" Jul 24 15:45:24.087493 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.087477 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 15:45:24.089891 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.089866 2569 fs.go:135] Filesystem UUIDs: map[61fb497b-8d52-49a8-903f-1471da3a19dc:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ebc50066-fc76-4918-a006-920d336944d4:/dev/nvme0n1p3] Jul 24 15:45:24.089970 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.089888 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 15:45:24.093629 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.093601 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 15:45:24.095998 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.095880 2569 manager.go:216] Machine: {Timestamp:2026-07-24 15:45:24.094602972 +0000 UTC m=+0.296330853 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100271 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2197dcc98df7f97c60651939c7d081 SystemUUID:ec2197dc-c98d-f7f9-7c60-651939c7d081 BootID:adbfbf09-8d52-4e68-aa4c-49759a987a94 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d7:96:8f:a9:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d7:96:8f:a9:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:01:8f:de:39:c1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 15:45:24.095998 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.095996 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 15:45:24.096123 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.096086 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 15:45:24.098609 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.098579 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 15:45:24.098753 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.098612 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-58.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 15:45:24.099734 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.099724 2569 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 15:45:24.099774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.099737 2569 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 15:45:24.099774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.099745 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 15:45:24.099774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.099759 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 15:45:24.100808 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.100798 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 15:45:24.102534 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.102522 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 24 15:45:24.102657 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.102647 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 15:45:24.102696 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.102677 2569 kubelet.go:486] "Attempting to sync node with API server" Jul 24 15:45:24.102696 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.102692 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 15:45:24.102757 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.102702 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 15:45:24.102757 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.102713 2569 kubelet.go:398] "Adding apiserver pod source" Jul 24 15:45:24.102757 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.102723 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 15:45:24.103239 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.103223 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 15:45:24.103323 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.103257 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 15:45:24.105979 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.105961 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 15:45:24.108707 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.108690 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 15:45:24.108774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.108721 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 15:45:24.110865 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.110844 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 15:45:24.110865 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.110869 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 15:45:24.110973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.110878 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 15:45:24.110973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.110886 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 15:45:24.110973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.110898 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 15:45:24.110973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.110905 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 15:45:24.110973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.110917 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 15:45:24.110973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.110924 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 15:45:24.110973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.110934 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 15:45:24.110973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.110941 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 15:45:24.110973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.110960 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 15:45:24.110973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.110965 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 15:45:24.111714 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.111701 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 15:45:24.111714 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.111712 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 15:45:24.113787 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:24.113758 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 15:45:24.113887 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:24.113758 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-58.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 15:45:24.115234 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.115222 2569 server.go:1268] "Started kubelet" Jul 24 15:45:24.115321 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.115295 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 15:45:24.115444 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.115410 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 15:45:24.115552 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.115476 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 15:45:24.115552 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.115474 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bb97s" Jul 24 15:45:24.116335 ip-10-0-141-58 systemd[1]: Started Kubernetes Kubelet. Jul 24 15:45:24.117296 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.117275 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 15:45:24.117930 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.117911 2569 server.go:310] "Adding debug handlers to kubelet server" Jul 24 15:45:24.121803 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:24.121752 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 15:45:24.123182 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.123165 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bb97s" Jul 24 15:45:24.124634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.124616 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-58.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 15:45:24.125299 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.125277 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 15:45:24.125299 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.125294 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 15:45:24.125519 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:24.125422 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found" Jul 24 15:45:24.125519 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:24.124618 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-58.ec2.internal.18c5444cd5aa67cf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-58.ec2.internal,UID:ip-10-0-141-58.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-58.ec2.internal,},FirstTimestamp:2026-07-24 15:45:24.115187663 +0000 UTC m=+0.316915543,LastTimestamp:2026-07-24 15:45:24.115187663 +0000 UTC m=+0.316915543,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-58.ec2.internal,}" Jul 24 15:45:24.125792 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.125778 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 15:45:24.125861 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.125805 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 15:45:24.125929 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.125918 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 15:45:24.126081 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.126070 2569 reconstruct.go:97] "Volume reconstruction finished" Jul 24 15:45:24.126081 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.126081 2569 reconciler.go:29] "Reconciler: start to sync state" Jul 24 15:45:24.126226 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.126204 2569 factory.go:55] Registering systemd factory Jul 24 15:45:24.126326 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.126230 2569 factory.go:223] Registration of the systemd container factory successfully Jul 24 15:45:24.126498 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.126481 2569 factory.go:153] Registering CRI-O factory Jul 24 15:45:24.126566 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.126504 2569 factory.go:223] Registration of the crio container factory successfully Jul 24 15:45:24.126623 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.126576 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 15:45:24.126623 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.126600 2569 factory.go:103] Registering Raw factory Jul 24 15:45:24.126623 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.126615 2569 manager.go:1196] Started watching for new ooms in manager Jul 24 15:45:24.127086 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.127073 2569 manager.go:349] Starting recovery of all containers Jul 24 15:45:24.129139 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:24.129111 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-58.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 15:45:24.129226 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:24.129132 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 15:45:24.139814 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.139787 2569 manager.go:354] Recovery completed Jul 24 15:45:24.145839 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.145823 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:45:24.147928 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.147910 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:45:24.148013 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.147954 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:45:24.148013 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.147970 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:45:24.148588 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.148569 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 15:45:24.148588 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.148580 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:45:24.148588 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.148585 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 15:45:24.148736 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.148605 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 24 15:45:24.149301 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.149283 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:45:24.149356 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.149311 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:45:24.149356 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.149324 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:45:24.150405 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.150390 2569 policy_none.go:49] "None policy: Start" Jul 24 15:45:24.150473 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.150409 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 15:45:24.150473 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.150420 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 15:45:24.151314 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:24.151303 2569 policy_none.go:47] "Start" Jul 24 15:45:25.476871 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.476847 2569 manager.go:337] "Starting Device Plugin manager" Jul 24 15:45:25.476991 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:25.476951 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 15:45:25.476991 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.476967 2569 server.go:87] "Starting device plugin registration server" Jul 24 15:45:25.477314 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.477295 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 15:45:25.477431 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.477315 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 15:45:25.477431 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.477422 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 15:45:25.477533 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.477515 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 15:45:25.477533 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.477525 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 15:45:25.478109 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:25.478088 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 15:45:25.478191 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:25.478136 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-58.ec2.internal\" not found" Jul 24 15:45:25.578504 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.578463 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:45:25.581685 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.581629 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:45:25.581685 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.581660 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:45:25.581685 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.581673 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:45:25.581855 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.581708 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-58.ec2.internal" Jul 24 15:45:25.589823 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.589805 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-58.ec2.internal" Jul 24 15:45:25.589908 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:25.589831 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-58.ec2.internal\": node \"ip-10-0-141-58.ec2.internal\" not found" Jul 24 15:45:25.604598 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:25.604575 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found" Jul 24 15:45:25.619389 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.619354 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 15:45:25.620760 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.620743 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 15:45:25.620836 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.620779 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 15:45:25.620836 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.620799 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 15:45:25.620905 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:25.620844 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 15:45:25.623318 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.623301 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:45:25.704961 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:25.704930 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found" Jul 24 15:45:25.721349 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.721326 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal"] Jul 24 15:45:25.721436 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.721403 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:45:25.722425 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.722407 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:45:25.722522 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.722438 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:45:25.722522 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.722449 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:45:25.723837 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.723824 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:45:25.723955 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.723932 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal" Jul 24 15:45:25.724008 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.723962 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:45:25.724646 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.724633 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:45:25.724729 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.724653 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:45:25.724729 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.724663 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:45:25.724729 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.724691 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:45:25.724729 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.724712 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:45:25.724729 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.724723 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:45:25.725892 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.725875 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal" Jul 24 15:45:25.725978 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.725900 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:45:25.726533 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.726519 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:45:25.726604 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.726537 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:45:25.726604 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.726546 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:45:25.750850 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:25.750823 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-58.ec2.internal\" not found" node="ip-10-0-141-58.ec2.internal" Jul 24 15:45:25.755177 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:25.755160 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-58.ec2.internal\" not found" node="ip-10-0-141-58.ec2.internal" Jul 24 15:45:25.805641 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:25.805613 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found" Jul 24 15:45:25.812969 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.812949 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f14b58b36cb1a9f322a5dc26c775590d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal\" (UID: \"f14b58b36cb1a9f322a5dc26c775590d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal" Jul 24 15:45:25.813067 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.812976 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f14b58b36cb1a9f322a5dc26c775590d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal\" (UID: \"f14b58b36cb1a9f322a5dc26c775590d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal" Jul 24 15:45:25.813067 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.812998 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dc55cd8cdabaf6971bfe9f07f04cbd9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-58.ec2.internal\" (UID: \"7dc55cd8cdabaf6971bfe9f07f04cbd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal" Jul 24 15:45:25.905891 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:25.905821 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found" Jul 24 15:45:25.914152 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.914114 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f14b58b36cb1a9f322a5dc26c775590d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal\" (UID: \"f14b58b36cb1a9f322a5dc26c775590d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal" Jul 24 15:45:25.914263 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.914170 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f14b58b36cb1a9f322a5dc26c775590d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal\" (UID: \"f14b58b36cb1a9f322a5dc26c775590d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal" Jul 24 15:45:25.914263 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.914187 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dc55cd8cdabaf6971bfe9f07f04cbd9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-58.ec2.internal\" (UID: \"7dc55cd8cdabaf6971bfe9f07f04cbd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal" Jul 24 15:45:25.914263 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.914239 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dc55cd8cdabaf6971bfe9f07f04cbd9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-58.ec2.internal\" (UID: \"7dc55cd8cdabaf6971bfe9f07f04cbd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal" Jul 24 15:45:25.914263 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.914249 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f14b58b36cb1a9f322a5dc26c775590d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal\" (UID: \"f14b58b36cb1a9f322a5dc26c775590d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal" Jul 24 15:45:25.914421 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:25.914254 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f14b58b36cb1a9f322a5dc26c775590d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal\" (UID: \"f14b58b36cb1a9f322a5dc26c775590d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal" Jul 24 15:45:26.006319 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:26.006284 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found" Jul 24 15:45:26.053715 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:26.053689 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal" Jul 24 15:45:26.059870 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:26.059854 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal" Jul 24 15:45:26.107076 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:26.107049 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found" Jul 24 15:45:26.207605 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:26.207527 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found" Jul 24 15:45:26.308095 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:26.308065 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found" Jul 24 15:45:26.350529 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:26.350498 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 15:45:26.351194 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:26.350639 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 15:45:26.408522 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:26.408491 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-58.ec2.internal\" not found" Jul 24 15:45:26.409629 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:26.409614 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:45:26.413227 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:26.413188 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 15:40:25 +0000 UTC" deadline="2028-02-23 04:57:02.929168479 +0000 UTC" Jul 24 15:45:26.413227 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:26.413224 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13885h11m36.5159479s" Jul 24 15:45:26.455422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:26.455390 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:45:26.509497 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:26.509423 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal" Jul 24 15:45:26.520537 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:26.520514 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 15:45:26.522232 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:26.522217 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal" Jul 24 15:45:26.531128 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:26.531110 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 15:45:26.628117 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:26.628073 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc55cd8cdabaf6971bfe9f07f04cbd9.slice/crio-6efd185ac9130b6176a3b8497505b826ae4b3dcf456c68b653ea4251b1a06769 WatchSource:0}: Error finding container 6efd185ac9130b6176a3b8497505b826ae4b3dcf456c68b653ea4251b1a06769: Status 404 returned error can't find the container with id 6efd185ac9130b6176a3b8497505b826ae4b3dcf456c68b653ea4251b1a06769 Jul 24 15:45:26.628510 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:26.628487 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14b58b36cb1a9f322a5dc26c775590d.slice/crio-cb281abfbbd4c3085afcfe03d883750407ffb93fbfde609b13f7b5d8dd846944 WatchSource:0}: Error finding container cb281abfbbd4c3085afcfe03d883750407ffb93fbfde609b13f7b5d8dd846944: Status 404 returned error can't find the container with id cb281abfbbd4c3085afcfe03d883750407ffb93fbfde609b13f7b5d8dd846944 Jul 24 15:45:26.634662 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:26.634646 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:45:26.776459 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:26.776386 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:45:27.388295 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.388026 2569 apiserver.go:52] "Watching apiserver" Jul 24 15:45:27.398313 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.398282 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 15:45:27.399475 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.399441 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-bdjkj","openshift-image-registry/node-ca-hvtww","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal","openshift-multus/multus-additional-cni-plugins-lkwbb","openshift-multus/network-metrics-daemon-pdlxm","openshift-network-diagnostics/network-check-target-tchwh","kube-system/global-pull-secret-syncer-lj6pt","kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx","openshift-multus/multus-d47kt","openshift-network-operator/iptables-alerter-txrqw","openshift-ovn-kubernetes/ovnkube-node-qm2n9","kube-system/konnectivity-agent-g2vqr","openshift-cluster-node-tuning-operator/tuned-6ztbq"] Jul 24 15:45:27.401505 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.401470 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:27.401624 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:27.401566 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lj6pt" podUID="d1348e8a-07ee-4ce2-afa4-2d580edd8df2" Jul 24 15:45:27.402984 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.402961 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g2vqr" Jul 24 15:45:27.404865 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.404848 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.405578 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.405539 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 15:45:27.405971 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.405813 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 15:45:27.405971 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.405847 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mmcmz\"" Jul 24 15:45:27.406989 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.406924 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.406989 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.406960 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:45:27.407381 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.407247 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m5xgj\"" Jul 24 15:45:27.408087 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.407846 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 15:45:27.408633 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.408373 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:27.408633 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:27.408446 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlxm" podUID="2ff63e8f-2348-417f-a83d-8e72db071ae7" Jul 24 15:45:27.409436 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.409418 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 15:45:27.409615 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.409598 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 15:45:27.410413 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.410280 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 15:45:27.410413 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.410289 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 15:45:27.410413 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.410308 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 15:45:27.410413 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.410282 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:27.410413 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:27.410370 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tchwh" podUID="2615ac1e-bb0a-4653-a612-f1406ea7d173" Jul 24 15:45:27.410413 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.410376 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 15:45:27.410413 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.410310 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c2bc7\"" Jul 24 15:45:27.411342 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.411323 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.412705 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.412686 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bdjkj" Jul 24 15:45:27.413576 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.413556 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 15:45:27.413849 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.413826 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 15:45:27.413947 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.413936 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zrnw6\"" Jul 24 15:45:27.414015 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.413997 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 15:45:27.414275 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.414260 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hvtww" Jul 24 15:45:27.415452 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.415434 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.418323 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.417382 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 15:45:27.418323 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.417635 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 15:45:27.418323 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.417711 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 15:45:27.418323 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.417816 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bmltr\"" Jul 24 15:45:27.418323 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.417895 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 15:45:27.418323 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.418077 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 15:45:27.418323 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.418274 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 15:45:27.420020 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.419103 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-22gft\"" Jul 24 15:45:27.420020 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.419201 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-txrqw" Jul 24 15:45:27.420524 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.420395 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-64b9x\"" Jul 24 15:45:27.420608 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.420592 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.421530 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.421504 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:45:27.421815 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.421794 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-57hwg\"" Jul 24 15:45:27.421953 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.421935 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 15:45:27.422013 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.421983 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 15:45:27.423768 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.423749 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 15:45:27.424076 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.424057 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 15:45:27.424318 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.424298 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5effb096-2376-43a2-83bd-4c3f3f085333-registration-dir\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.425774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.424655 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5effb096-2376-43a2-83bd-4c3f3f085333-device-dir\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.425774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.424691 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk88g\" (UniqueName: \"kubernetes.io/projected/5effb096-2376-43a2-83bd-4c3f3f085333-kube-api-access-wk88g\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.425774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.424847 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-sys\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.425774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.424873 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxrcq\" (UniqueName: \"kubernetes.io/projected/c1fd1def-d92a-438b-a6a1-0fec4b699fea-kube-api-access-wxrcq\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.425774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.424878 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 15:45:27.425774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.424899 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5effb096-2376-43a2-83bd-4c3f3f085333-socket-dir\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.425774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.424941 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-modprobe-d\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.425774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.424963 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-run\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.425774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425031 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd7f5\" (UniqueName: \"kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5\") pod \"network-check-target-tchwh\" (UID: \"2615ac1e-bb0a-4653-a612-f1406ea7d173\") " pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:27.425774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425076 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs\") pod \"network-metrics-daemon-pdlxm\" (UID: \"2ff63e8f-2348-417f-a83d-8e72db071ae7\") " pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:27.425774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425094 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 15:45:27.425774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425104 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f58d4fe-4a3c-466c-8001-85fd8797a32a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.425774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425144 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret\") pod \"global-pull-secret-syncer-lj6pt\" (UID: \"d1348e8a-07ee-4ce2-afa4-2d580edd8df2\") " pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:27.425774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425170 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6f58d4fe-4a3c-466c-8001-85fd8797a32a-os-release\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.425774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425192 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5effb096-2376-43a2-83bd-4c3f3f085333-sys-fs\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.425774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425245 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-sysconfig\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.425774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425268 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-sysctl-conf\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.426774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425290 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f58d4fe-4a3c-466c-8001-85fd8797a32a-cni-binary-copy\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.426774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425312 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6f58d4fe-4a3c-466c-8001-85fd8797a32a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.426774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425338 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-kubelet-config\") pod \"global-pull-secret-syncer-lj6pt\" (UID: \"d1348e8a-07ee-4ce2-afa4-2d580edd8df2\") " pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:27.426774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425359 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/357d493f-b503-4d18-b37c-e8e7168f84c2-konnectivity-ca\") pod \"konnectivity-agent-g2vqr\" (UID: \"357d493f-b503-4d18-b37c-e8e7168f84c2\") " pod="kube-system/konnectivity-agent-g2vqr" Jul 24 15:45:27.426774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.424488 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 15:45:27.426774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425380 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-var-lib-kubelet\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.426774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425403 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjrxp\" (UniqueName: \"kubernetes.io/projected/6f58d4fe-4a3c-466c-8001-85fd8797a32a-kube-api-access-pjrxp\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.426774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425442 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5effb096-2376-43a2-83bd-4c3f3f085333-etc-selinux\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.426774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425467 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-sysctl-d\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.426774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425531 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1fd1def-d92a-438b-a6a1-0fec4b699fea-tmp\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.426774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425052 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 15:45:27.426774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.424537 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 15:45:27.426774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425579 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99p85\" (UniqueName: \"kubernetes.io/projected/2ff63e8f-2348-417f-a83d-8e72db071ae7-kube-api-access-99p85\") pod \"network-metrics-daemon-pdlxm\" (UID: \"2ff63e8f-2348-417f-a83d-8e72db071ae7\") " pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:27.426774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425614 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5effb096-2376-43a2-83bd-4c3f3f085333-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.426774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.424816 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pc56q\"" Jul 24 15:45:27.426774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425679 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-dbus\") pod \"global-pull-secret-syncer-lj6pt\" (UID: \"d1348e8a-07ee-4ce2-afa4-2d580edd8df2\") " pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:27.426774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425745 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/357d493f-b503-4d18-b37c-e8e7168f84c2-agent-certs\") pod \"konnectivity-agent-g2vqr\" (UID: \"357d493f-b503-4d18-b37c-e8e7168f84c2\") " pod="kube-system/konnectivity-agent-g2vqr" Jul 24 15:45:27.426774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425784 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-host\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.427564 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425847 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-tuned\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.427564 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425896 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f58d4fe-4a3c-466c-8001-85fd8797a32a-system-cni-dir\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.427564 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425936 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f58d4fe-4a3c-466c-8001-85fd8797a32a-cnibin\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.427564 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425969 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6f58d4fe-4a3c-466c-8001-85fd8797a32a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.427564 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.425996 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-kubernetes\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.427564 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.426048 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-systemd\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.427564 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.426079 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-lib-modules\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.427564 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.426169 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:45:27.472012 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.471986 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-86b2w" Jul 24 15:45:27.476391 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.476366 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-86b2w" Jul 24 15:45:27.511109 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.511054 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 15:45:27.526525 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526500 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-modprobe-d\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.526663 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526528 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-run\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.526663 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526552 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-host-run-netns\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.526663 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526571 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.526663 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526593 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/34e569a7-61a3-4909-865f-36a7bd286efa-serviceca\") pod \"node-ca-hvtww\" (UID: \"34e569a7-61a3-4909-865f-36a7bd286efa\") " pod="openshift-image-registry/node-ca-hvtww" Jul 24 15:45:27.526663 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526617 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f58d4fe-4a3c-466c-8001-85fd8797a32a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.526663 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526639 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-run\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.526948 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526643 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret\") pod \"global-pull-secret-syncer-lj6pt\" (UID: \"d1348e8a-07ee-4ce2-afa4-2d580edd8df2\") " pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:27.526948 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526685 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-modprobe-d\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.526948 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526699 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6f58d4fe-4a3c-466c-8001-85fd8797a32a-os-release\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.526948 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526730 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-multus-socket-dir-parent\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.526948 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:27.526738 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:45:27.526948 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526758 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-multus-conf-dir\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.526948 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526768 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f58d4fe-4a3c-466c-8001-85fd8797a32a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.526948 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526786 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnj6d\" (UniqueName: \"kubernetes.io/projected/0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8-kube-api-access-gnj6d\") pod \"iptables-alerter-txrqw\" (UID: \"0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8\") " pod="openshift-network-operator/iptables-alerter-txrqw" Jul 24 15:45:27.526948 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526790 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6f58d4fe-4a3c-466c-8001-85fd8797a32a-os-release\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.526948 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:27.526833 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret podName:d1348e8a-07ee-4ce2-afa4-2d580edd8df2 nodeName:}" failed. No retries permitted until 2026-07-24 15:45:28.026792575 +0000 UTC m=+2.944762872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret") pod "global-pull-secret-syncer-lj6pt" (UID: "d1348e8a-07ee-4ce2-afa4-2d580edd8df2") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:45:27.526948 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526879 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-sysctl-conf\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.526948 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526903 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-os-release\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.526948 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526926 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/43d9054e-c012-47be-8143-968de45e5b9b-multus-daemon-config\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.526948 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526949 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-run-ovn-kubernetes\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.527687 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526972 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c4r4\" (UniqueName: \"kubernetes.io/projected/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-kube-api-access-7c4r4\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.527687 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.526993 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3447e518-ab20-411f-aa9f-4918b9246385-hosts-file\") pod \"node-resolver-bdjkj\" (UID: \"3447e518-ab20-411f-aa9f-4918b9246385\") " pod="openshift-dns/node-resolver-bdjkj" Jul 24 15:45:27.527687 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527031 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-kubelet-config\") pod \"global-pull-secret-syncer-lj6pt\" (UID: \"d1348e8a-07ee-4ce2-afa4-2d580edd8df2\") " pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:27.527687 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527063 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/357d493f-b503-4d18-b37c-e8e7168f84c2-konnectivity-ca\") pod \"konnectivity-agent-g2vqr\" (UID: \"357d493f-b503-4d18-b37c-e8e7168f84c2\") " pod="kube-system/konnectivity-agent-g2vqr" Jul 24 15:45:27.527687 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527089 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjrxp\" (UniqueName: \"kubernetes.io/projected/6f58d4fe-4a3c-466c-8001-85fd8797a32a-kube-api-access-pjrxp\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.527687 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527032 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-sysctl-conf\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.527687 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527116 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5effb096-2376-43a2-83bd-4c3f3f085333-etc-selinux\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.527687 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527142 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-run-systemd\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.527687 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527156 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-kubelet-config\") pod \"global-pull-secret-syncer-lj6pt\" (UID: \"d1348e8a-07ee-4ce2-afa4-2d580edd8df2\") " pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:27.527687 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527166 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjm8v\" (UniqueName: \"kubernetes.io/projected/3447e518-ab20-411f-aa9f-4918b9246385-kube-api-access-hjm8v\") pod \"node-resolver-bdjkj\" (UID: \"3447e518-ab20-411f-aa9f-4918b9246385\") " pod="openshift-dns/node-resolver-bdjkj" Jul 24 15:45:27.527687 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527191 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-sysctl-d\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.527687 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527260 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1fd1def-d92a-438b-a6a1-0fec4b699fea-tmp\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.527687 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527452 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5effb096-2376-43a2-83bd-4c3f3f085333-etc-selinux\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.527687 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527517 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99p85\" (UniqueName: \"kubernetes.io/projected/2ff63e8f-2348-417f-a83d-8e72db071ae7-kube-api-access-99p85\") pod \"network-metrics-daemon-pdlxm\" (UID: \"2ff63e8f-2348-417f-a83d-8e72db071ae7\") " pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:27.527687 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527583 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5effb096-2376-43a2-83bd-4c3f3f085333-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.527687 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527615 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-host-run-k8s-cni-cncf-io\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.527687 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527651 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5effb096-2376-43a2-83bd-4c3f3f085333-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.528499 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527669 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 15:45:27.528499 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527765 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-sysctl-d\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.528499 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527798 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-dbus\") pod \"global-pull-secret-syncer-lj6pt\" (UID: \"d1348e8a-07ee-4ce2-afa4-2d580edd8df2\") " pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:27.528499 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527836 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/357d493f-b503-4d18-b37c-e8e7168f84c2-agent-certs\") pod \"konnectivity-agent-g2vqr\" (UID: \"357d493f-b503-4d18-b37c-e8e7168f84c2\") " pod="kube-system/konnectivity-agent-g2vqr" Jul 24 15:45:27.528499 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527861 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f58d4fe-4a3c-466c-8001-85fd8797a32a-system-cni-dir\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.528499 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527884 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f58d4fe-4a3c-466c-8001-85fd8797a32a-cnibin\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.528499 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527898 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-dbus\") pod \"global-pull-secret-syncer-lj6pt\" (UID: \"d1348e8a-07ee-4ce2-afa4-2d580edd8df2\") " pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:27.528499 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527936 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f58d4fe-4a3c-466c-8001-85fd8797a32a-system-cni-dir\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.528499 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527836 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/357d493f-b503-4d18-b37c-e8e7168f84c2-konnectivity-ca\") pod \"konnectivity-agent-g2vqr\" (UID: \"357d493f-b503-4d18-b37c-e8e7168f84c2\") " pod="kube-system/konnectivity-agent-g2vqr" Jul 24 15:45:27.528499 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527993 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f58d4fe-4a3c-466c-8001-85fd8797a32a-cnibin\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.528499 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.527910 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6f58d4fe-4a3c-466c-8001-85fd8797a32a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.528499 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528040 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk88g\" (UniqueName: \"kubernetes.io/projected/5effb096-2376-43a2-83bd-4c3f3f085333-kube-api-access-wk88g\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.528499 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528078 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-node-log\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.528499 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528102 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5effb096-2376-43a2-83bd-4c3f3f085333-registration-dir\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.528499 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528128 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-multus-cni-dir\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.528499 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528158 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-cnibin\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.528499 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528181 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43d9054e-c012-47be-8143-968de45e5b9b-cni-binary-copy\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.529273 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528220 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-host-var-lib-cni-bin\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.529273 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528252 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmrrf\" (UniqueName: \"kubernetes.io/projected/43d9054e-c012-47be-8143-968de45e5b9b-kube-api-access-hmrrf\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.529273 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528274 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-systemd-units\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.529273 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528296 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-etc-openvswitch\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.529273 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528320 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxrcq\" (UniqueName: \"kubernetes.io/projected/c1fd1def-d92a-438b-a6a1-0fec4b699fea-kube-api-access-wxrcq\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.529273 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528346 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5effb096-2376-43a2-83bd-4c3f3f085333-socket-dir\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.529273 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528369 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-ovn-node-metrics-cert\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.529273 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528395 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-ovnkube-script-lib\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.529273 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528418 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3447e518-ab20-411f-aa9f-4918b9246385-tmp-dir\") pod \"node-resolver-bdjkj\" (UID: \"3447e518-ab20-411f-aa9f-4918b9246385\") " pod="openshift-dns/node-resolver-bdjkj" Jul 24 15:45:27.529273 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528447 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd7f5\" (UniqueName: \"kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5\") pod \"network-check-target-tchwh\" (UID: \"2615ac1e-bb0a-4653-a612-f1406ea7d173\") " pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:27.529273 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528473 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs\") pod \"network-metrics-daemon-pdlxm\" (UID: \"2ff63e8f-2348-417f-a83d-8e72db071ae7\") " pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:27.529273 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528498 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-kubelet\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.529273 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528523 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-run-openvswitch\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.529273 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528547 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-cni-netd\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.529273 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528570 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34e569a7-61a3-4909-865f-36a7bd286efa-host\") pod \"node-ca-hvtww\" (UID: \"34e569a7-61a3-4909-865f-36a7bd286efa\") " pod="openshift-image-registry/node-ca-hvtww" Jul 24 15:45:27.529273 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528570 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6f58d4fe-4a3c-466c-8001-85fd8797a32a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.530004 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528594 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5effb096-2376-43a2-83bd-4c3f3f085333-sys-fs\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.530004 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528617 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-system-cni-dir\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.530004 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528642 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-run-ovn\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.530004 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528666 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-sysconfig\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.530004 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528690 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f58d4fe-4a3c-466c-8001-85fd8797a32a-cni-binary-copy\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.530004 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528723 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6f58d4fe-4a3c-466c-8001-85fd8797a32a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.530004 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528748 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-var-lib-openvswitch\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.530004 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528769 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-log-socket\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.530004 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528798 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-ovnkube-config\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.530004 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528869 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5effb096-2376-43a2-83bd-4c3f3f085333-sys-fs\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.530004 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528908 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-var-lib-kubelet\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.530004 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528937 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-sysconfig\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.530004 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528958 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-run-netns\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.530004 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528985 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-cni-bin\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.530004 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.528987 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5effb096-2376-43a2-83bd-4c3f3f085333-registration-dir\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.530004 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529036 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-env-overrides\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.530004 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529062 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-var-lib-kubelet\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.530776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529082 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnczs\" (UniqueName: \"kubernetes.io/projected/34e569a7-61a3-4909-865f-36a7bd286efa-kube-api-access-nnczs\") pod \"node-ca-hvtww\" (UID: \"34e569a7-61a3-4909-865f-36a7bd286efa\") " pod="openshift-image-registry/node-ca-hvtww" Jul 24 15:45:27.530776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529112 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-host-var-lib-cni-multus\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.530776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529137 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-hostroot\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.530776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529161 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-host-run-multus-certs\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.530776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529185 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-etc-kubernetes\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.530776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529231 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-host\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.530776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529291 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-tuned\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.530776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529320 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8-host-slash\") pod \"iptables-alerter-txrqw\" (UID: \"0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8\") " pod="openshift-network-operator/iptables-alerter-txrqw" Jul 24 15:45:27.530776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529346 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-kubernetes\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.530776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529371 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-systemd\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.530776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529398 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-lib-modules\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.530776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529419 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f58d4fe-4a3c-466c-8001-85fd8797a32a-cni-binary-copy\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.530776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529437 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5effb096-2376-43a2-83bd-4c3f3f085333-device-dir\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.530776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529462 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-host-var-lib-kubelet\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.530776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529489 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8-iptables-alerter-script\") pod \"iptables-alerter-txrqw\" (UID: \"0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8\") " pod="openshift-network-operator/iptables-alerter-txrqw" Jul 24 15:45:27.530776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529514 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-slash\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.530776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529540 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-sys\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.531571 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529578 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6f58d4fe-4a3c-466c-8001-85fd8797a32a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.531571 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529669 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-host\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.531571 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:27.529732 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:45:27.531571 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529740 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5effb096-2376-43a2-83bd-4c3f3f085333-socket-dir\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.531571 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:27.529780 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs podName:2ff63e8f-2348-417f-a83d-8e72db071ae7 nodeName:}" failed. No retries permitted until 2026-07-24 15:45:28.029763573 +0000 UTC m=+2.947733866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs") pod "network-metrics-daemon-pdlxm" (UID: "2ff63e8f-2348-417f-a83d-8e72db071ae7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:45:27.531571 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529821 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-lib-modules\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.531571 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529868 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5effb096-2376-43a2-83bd-4c3f3f085333-device-dir\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.531571 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529861 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-systemd\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.531571 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529821 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-sys\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.531571 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.529909 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-kubernetes\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.532047 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.531633 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/357d493f-b503-4d18-b37c-e8e7168f84c2-agent-certs\") pod \"konnectivity-agent-g2vqr\" (UID: \"357d493f-b503-4d18-b37c-e8e7168f84c2\") " pod="kube-system/konnectivity-agent-g2vqr" Jul 24 15:45:27.532466 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.532271 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1fd1def-d92a-438b-a6a1-0fec4b699fea-tmp\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.532873 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.532852 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1fd1def-d92a-438b-a6a1-0fec4b699fea-etc-tuned\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.536437 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.536392 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99p85\" (UniqueName: \"kubernetes.io/projected/2ff63e8f-2348-417f-a83d-8e72db071ae7-kube-api-access-99p85\") pod \"network-metrics-daemon-pdlxm\" (UID: \"2ff63e8f-2348-417f-a83d-8e72db071ae7\") " pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:27.537350 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.536947 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjrxp\" (UniqueName: \"kubernetes.io/projected/6f58d4fe-4a3c-466c-8001-85fd8797a32a-kube-api-access-pjrxp\") pod \"multus-additional-cni-plugins-lkwbb\" (UID: \"6f58d4fe-4a3c-466c-8001-85fd8797a32a\") " pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.537978 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:27.537838 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:45:27.537978 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:27.537867 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:45:27.537978 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:27.537882 2569 projected.go:196] Error preparing data for projected volume kube-api-access-bd7f5 for pod openshift-network-diagnostics/network-check-target-tchwh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:45:27.537978 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.537924 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk88g\" (UniqueName: \"kubernetes.io/projected/5effb096-2376-43a2-83bd-4c3f3f085333-kube-api-access-wk88g\") pod \"aws-ebs-csi-driver-node-rlbpx\" (UID: \"5effb096-2376-43a2-83bd-4c3f3f085333\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.537978 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:27.537953 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5 podName:2615ac1e-bb0a-4653-a612-f1406ea7d173 nodeName:}" failed. No retries permitted until 2026-07-24 15:45:28.037919121 +0000 UTC m=+2.955889399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bd7f5" (UniqueName: "kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5") pod "network-check-target-tchwh" (UID: "2615ac1e-bb0a-4653-a612-f1406ea7d173") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:45:27.538376 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.538352 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxrcq\" (UniqueName: \"kubernetes.io/projected/c1fd1def-d92a-438b-a6a1-0fec4b699fea-kube-api-access-wxrcq\") pod \"tuned-6ztbq\" (UID: \"c1fd1def-d92a-438b-a6a1-0fec4b699fea\") " pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.627583 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.626900 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal" event={"ID":"7dc55cd8cdabaf6971bfe9f07f04cbd9","Type":"ContainerStarted","Data":"6efd185ac9130b6176a3b8497505b826ae4b3dcf456c68b653ea4251b1a06769"} Jul 24 15:45:27.630158 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630131 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-multus-conf-dir\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.630330 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630169 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnj6d\" (UniqueName: \"kubernetes.io/projected/0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8-kube-api-access-gnj6d\") pod \"iptables-alerter-txrqw\" (UID: \"0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8\") " pod="openshift-network-operator/iptables-alerter-txrqw" Jul 24 15:45:27.630330 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630197 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-os-release\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.630330 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630237 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/43d9054e-c012-47be-8143-968de45e5b9b-multus-daemon-config\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.630330 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630254 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-multus-conf-dir\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.630330 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630262 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-run-ovn-kubernetes\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.630330 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630301 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-run-ovn-kubernetes\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.630330 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630304 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c4r4\" (UniqueName: \"kubernetes.io/projected/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-kube-api-access-7c4r4\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.630648 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630341 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3447e518-ab20-411f-aa9f-4918b9246385-hosts-file\") pod \"node-resolver-bdjkj\" (UID: \"3447e518-ab20-411f-aa9f-4918b9246385\") " pod="openshift-dns/node-resolver-bdjkj" Jul 24 15:45:27.630648 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630367 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-run-systemd\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.630648 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630393 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjm8v\" (UniqueName: \"kubernetes.io/projected/3447e518-ab20-411f-aa9f-4918b9246385-kube-api-access-hjm8v\") pod \"node-resolver-bdjkj\" (UID: \"3447e518-ab20-411f-aa9f-4918b9246385\") " pod="openshift-dns/node-resolver-bdjkj" Jul 24 15:45:27.630648 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630419 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-host-run-k8s-cni-cncf-io\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.630648 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630446 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-node-log\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.630648 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630469 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-multus-cni-dir\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.630648 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630491 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-cnibin\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.630648 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630505 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43d9054e-c012-47be-8143-968de45e5b9b-cni-binary-copy\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.630648 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630522 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-run-systemd\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.630648 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630538 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-node-log\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.630648 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630526 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-host-var-lib-cni-bin\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.630648 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630554 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-host-var-lib-cni-bin\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.630648 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630584 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-os-release\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.630648 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630591 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmrrf\" (UniqueName: \"kubernetes.io/projected/43d9054e-c012-47be-8143-968de45e5b9b-kube-api-access-hmrrf\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.630648 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630613 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3447e518-ab20-411f-aa9f-4918b9246385-hosts-file\") pod \"node-resolver-bdjkj\" (UID: \"3447e518-ab20-411f-aa9f-4918b9246385\") " pod="openshift-dns/node-resolver-bdjkj" Jul 24 15:45:27.630648 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630651 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-systemd-units\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.631422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630617 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-systemd-units\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.631422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630687 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-etc-openvswitch\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.631422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630698 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-host-run-k8s-cni-cncf-io\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.631422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630734 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-ovn-node-metrics-cert\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.631422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630761 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-ovnkube-script-lib\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.631422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630784 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3447e518-ab20-411f-aa9f-4918b9246385-tmp-dir\") pod \"node-resolver-bdjkj\" (UID: \"3447e518-ab20-411f-aa9f-4918b9246385\") " pod="openshift-dns/node-resolver-bdjkj" Jul 24 15:45:27.631422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630815 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-etc-openvswitch\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.631422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630838 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-kubelet\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.631422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630863 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-run-openvswitch\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.631422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630886 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-cni-netd\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.631422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630908 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34e569a7-61a3-4909-865f-36a7bd286efa-host\") pod \"node-ca-hvtww\" (UID: \"34e569a7-61a3-4909-865f-36a7bd286efa\") " pod="openshift-image-registry/node-ca-hvtww" Jul 24 15:45:27.631422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630927 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal" event={"ID":"f14b58b36cb1a9f322a5dc26c775590d","Type":"ContainerStarted","Data":"cb281abfbbd4c3085afcfe03d883750407ffb93fbfde609b13f7b5d8dd846944"} Jul 24 15:45:27.631422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630931 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-system-cni-dir\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.631422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630983 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-system-cni-dir\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.631422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630991 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-run-ovn\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.631422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.630780 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-multus-cni-dir\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.631422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631021 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-var-lib-openvswitch\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.631422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631043 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-kubelet\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.632133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631047 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-log-socket\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.632133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631074 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-ovnkube-config\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.632133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631092 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-cni-netd\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.632133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631099 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-run-netns\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.632133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631120 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3447e518-ab20-411f-aa9f-4918b9246385-tmp-dir\") pod \"node-resolver-bdjkj\" (UID: \"3447e518-ab20-411f-aa9f-4918b9246385\") " pod="openshift-dns/node-resolver-bdjkj" Jul 24 15:45:27.632133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631126 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-cni-bin\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.632133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631137 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/34e569a7-61a3-4909-865f-36a7bd286efa-host\") pod \"node-ca-hvtww\" (UID: \"34e569a7-61a3-4909-865f-36a7bd286efa\") " pod="openshift-image-registry/node-ca-hvtww" Jul 24 15:45:27.632133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631155 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-env-overrides\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.632133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631165 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-cni-bin\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.632133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631176 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnczs\" (UniqueName: \"kubernetes.io/projected/34e569a7-61a3-4909-865f-36a7bd286efa-kube-api-access-nnczs\") pod \"node-ca-hvtww\" (UID: \"34e569a7-61a3-4909-865f-36a7bd286efa\") " pod="openshift-image-registry/node-ca-hvtww" Jul 24 15:45:27.632133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631192 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-host-var-lib-cni-multus\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.632133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631225 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-hostroot\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.632133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631247 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/43d9054e-c012-47be-8143-968de45e5b9b-cni-binary-copy\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.632133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631274 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-host-run-multus-certs\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.632133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631253 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-var-lib-openvswitch\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.632133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631280 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/43d9054e-c012-47be-8143-968de45e5b9b-multus-daemon-config\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.632133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631292 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-etc-kubernetes\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.632133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631312 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-run-openvswitch\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.632725 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631325 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8-host-slash\") pod \"iptables-alerter-txrqw\" (UID: \"0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8\") " pod="openshift-network-operator/iptables-alerter-txrqw" Jul 24 15:45:27.632725 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631352 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-run-netns\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.632725 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631355 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-host-var-lib-kubelet\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.632725 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631384 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8-iptables-alerter-script\") pod \"iptables-alerter-txrqw\" (UID: \"0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8\") " pod="openshift-network-operator/iptables-alerter-txrqw" Jul 24 15:45:27.632725 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631390 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-host-var-lib-kubelet\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.632725 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631408 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-slash\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.632725 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631435 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-host-run-netns\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.632725 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631446 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-cnibin\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.632725 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631283 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-log-socket\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.632725 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631460 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.632725 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631487 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/34e569a7-61a3-4909-865f-36a7bd286efa-serviceca\") pod \"node-ca-hvtww\" (UID: \"34e569a7-61a3-4909-865f-36a7bd286efa\") " pod="openshift-image-registry/node-ca-hvtww" Jul 24 15:45:27.632725 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631489 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8-host-slash\") pod \"iptables-alerter-txrqw\" (UID: \"0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8\") " pod="openshift-network-operator/iptables-alerter-txrqw" Jul 24 15:45:27.632725 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631515 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-host-var-lib-cni-multus\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.632725 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631206 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-run-ovn\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.632725 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631540 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-multus-socket-dir-parent\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.632725 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631545 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-etc-kubernetes\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.632725 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631533 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-slash\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.632725 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631579 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-host-run-multus-certs\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.633325 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631609 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-hostroot\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.633325 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631645 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.633325 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631691 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-multus-socket-dir-parent\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.633325 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631729 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/43d9054e-c012-47be-8143-968de45e5b9b-host-run-netns\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.633325 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.631989 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-env-overrides\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.633325 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.632043 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/34e569a7-61a3-4909-865f-36a7bd286efa-serviceca\") pod \"node-ca-hvtww\" (UID: \"34e569a7-61a3-4909-865f-36a7bd286efa\") " pod="openshift-image-registry/node-ca-hvtww" Jul 24 15:45:27.633325 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.632090 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8-iptables-alerter-script\") pod \"iptables-alerter-txrqw\" (UID: \"0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8\") " pod="openshift-network-operator/iptables-alerter-txrqw" Jul 24 15:45:27.633325 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.632694 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-ovnkube-config\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.633325 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.632709 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-ovnkube-script-lib\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.633751 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.633730 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-ovn-node-metrics-cert\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.638490 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.638432 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnj6d\" (UniqueName: \"kubernetes.io/projected/0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8-kube-api-access-gnj6d\") pod \"iptables-alerter-txrqw\" (UID: \"0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8\") " pod="openshift-network-operator/iptables-alerter-txrqw" Jul 24 15:45:27.638650 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.638630 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c4r4\" (UniqueName: \"kubernetes.io/projected/fc74e7b2-f0e1-4fe3-b02a-567e2f726143-kube-api-access-7c4r4\") pod \"ovnkube-node-qm2n9\" (UID: \"fc74e7b2-f0e1-4fe3-b02a-567e2f726143\") " pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:27.638704 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.638643 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmrrf\" (UniqueName: \"kubernetes.io/projected/43d9054e-c012-47be-8143-968de45e5b9b-kube-api-access-hmrrf\") pod \"multus-d47kt\" (UID: \"43d9054e-c012-47be-8143-968de45e5b9b\") " pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.639348 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.639331 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjm8v\" (UniqueName: \"kubernetes.io/projected/3447e518-ab20-411f-aa9f-4918b9246385-kube-api-access-hjm8v\") pod \"node-resolver-bdjkj\" (UID: \"3447e518-ab20-411f-aa9f-4918b9246385\") " pod="openshift-dns/node-resolver-bdjkj" Jul 24 15:45:27.641947 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.641930 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnczs\" (UniqueName: \"kubernetes.io/projected/34e569a7-61a3-4909-865f-36a7bd286efa-kube-api-access-nnczs\") pod \"node-ca-hvtww\" (UID: \"34e569a7-61a3-4909-865f-36a7bd286efa\") " pod="openshift-image-registry/node-ca-hvtww" Jul 24 15:45:27.725446 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.725410 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g2vqr" Jul 24 15:45:27.729093 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.729070 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" Jul 24 15:45:27.741149 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.741126 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lkwbb" Jul 24 15:45:27.746818 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.746798 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" Jul 24 15:45:27.753581 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.753559 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bdjkj" Jul 24 15:45:27.763451 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.763434 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hvtww" Jul 24 15:45:27.771046 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.771025 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d47kt" Jul 24 15:45:27.779542 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.779520 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-txrqw" Jul 24 15:45:27.788354 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:27.788332 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:28.033362 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:28.033270 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret\") pod \"global-pull-secret-syncer-lj6pt\" (UID: \"d1348e8a-07ee-4ce2-afa4-2d580edd8df2\") " pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:28.033512 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:28.033443 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs\") pod \"network-metrics-daemon-pdlxm\" (UID: \"2ff63e8f-2348-417f-a83d-8e72db071ae7\") " pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:28.033512 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:28.033453 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:45:28.033618 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:28.033538 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret podName:d1348e8a-07ee-4ce2-afa4-2d580edd8df2 nodeName:}" failed. No retries permitted until 2026-07-24 15:45:29.033515209 +0000 UTC m=+3.951485496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret") pod "global-pull-secret-syncer-lj6pt" (UID: "d1348e8a-07ee-4ce2-afa4-2d580edd8df2") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:45:28.033618 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:28.033601 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:45:28.033724 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:28.033668 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs podName:2ff63e8f-2348-417f-a83d-8e72db071ae7 nodeName:}" failed. No retries permitted until 2026-07-24 15:45:29.033648336 +0000 UTC m=+3.951618630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs") pod "network-metrics-daemon-pdlxm" (UID: "2ff63e8f-2348-417f-a83d-8e72db071ae7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:45:28.134594 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:28.134556 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd7f5\" (UniqueName: \"kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5\") pod \"network-check-target-tchwh\" (UID: \"2615ac1e-bb0a-4653-a612-f1406ea7d173\") " pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:28.134791 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:28.134698 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:45:28.134791 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:28.134712 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:45:28.134791 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:28.134723 2569 projected.go:196] Error preparing data for projected volume kube-api-access-bd7f5 for pod openshift-network-diagnostics/network-check-target-tchwh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:45:28.134791 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:28.134771 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5 podName:2615ac1e-bb0a-4653-a612-f1406ea7d173 nodeName:}" failed. No retries permitted until 2026-07-24 15:45:29.134758424 +0000 UTC m=+4.052728701 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bd7f5" (UniqueName: "kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5") pod "network-check-target-tchwh" (UID: "2615ac1e-bb0a-4653-a612-f1406ea7d173") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:45:28.479370 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:28.479333 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 15:40:27 +0000 UTC" deadline="2028-03-17 13:46:30.233616698 +0000 UTC" Jul 24 15:45:28.479370 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:28.479363 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14446h1m1.754257085s" Jul 24 15:45:28.489708 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:28.489539 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f58d4fe_4a3c_466c_8001_85fd8797a32a.slice/crio-dcfde2d9411ce266e07f1e5097aa54d5489715033b7c433967fe6a295e5c3446 WatchSource:0}: Error finding container dcfde2d9411ce266e07f1e5097aa54d5489715033b7c433967fe6a295e5c3446: Status 404 returned error can't find the container with id dcfde2d9411ce266e07f1e5097aa54d5489715033b7c433967fe6a295e5c3446 Jul 24 15:45:28.491048 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:28.491023 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc74e7b2_f0e1_4fe3_b02a_567e2f726143.slice/crio-fca208f664688c00ee1141abe2f71e2b8eead5d369b30b3d12e9cc172aa04e0d WatchSource:0}: Error finding container fca208f664688c00ee1141abe2f71e2b8eead5d369b30b3d12e9cc172aa04e0d: Status 404 returned error can't find the container with id fca208f664688c00ee1141abe2f71e2b8eead5d369b30b3d12e9cc172aa04e0d Jul 24 15:45:28.495282 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:28.495159 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d9054e_c012_47be_8143_968de45e5b9b.slice/crio-6e6da93ff9c3de2e44cd727261b3b7d9c2cbf570f508d06fe2dfee3f2fbffa4b WatchSource:0}: Error finding container 6e6da93ff9c3de2e44cd727261b3b7d9c2cbf570f508d06fe2dfee3f2fbffa4b: Status 404 returned error can't find the container with id 6e6da93ff9c3de2e44cd727261b3b7d9c2cbf570f508d06fe2dfee3f2fbffa4b Jul 24 15:45:28.497676 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:28.497655 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e569a7_61a3_4909_865f_36a7bd286efa.slice/crio-0434834c2454e5501cfe021a598ae3fb8935ab55d0016f3549b74d2c6923a136 WatchSource:0}: Error finding container 0434834c2454e5501cfe021a598ae3fb8935ab55d0016f3549b74d2c6923a136: Status 404 returned error can't find the container with id 0434834c2454e5501cfe021a598ae3fb8935ab55d0016f3549b74d2c6923a136 Jul 24 15:45:28.499342 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:28.499322 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5effb096_2376_43a2_83bd_4c3f3f085333.slice/crio-4baade4036db7841ff68f2892196d45f51e1f10efeaeffc8a10e919fb03284c3 WatchSource:0}: Error finding container 4baade4036db7841ff68f2892196d45f51e1f10efeaeffc8a10e919fb03284c3: Status 404 returned error can't find the container with id 4baade4036db7841ff68f2892196d45f51e1f10efeaeffc8a10e919fb03284c3 Jul 24 15:45:28.500849 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:28.500824 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5f0ba5_21bf_4de8_9b8b_3f09ad6e5bc8.slice/crio-3e74b988683e2c8e3b90671262bd9c3a5ed39b65f9620d3ce987ebafda19d47c WatchSource:0}: Error finding container 3e74b988683e2c8e3b90671262bd9c3a5ed39b65f9620d3ce987ebafda19d47c: Status 404 returned error can't find the container with id 3e74b988683e2c8e3b90671262bd9c3a5ed39b65f9620d3ce987ebafda19d47c Jul 24 15:45:28.501869 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:28.501842 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3447e518_ab20_411f_aa9f_4918b9246385.slice/crio-4443843202d3ada16d5f1476f9c944b24135edef493473b8b4bdda3c9d540658 WatchSource:0}: Error finding container 4443843202d3ada16d5f1476f9c944b24135edef493473b8b4bdda3c9d540658: Status 404 returned error can't find the container with id 4443843202d3ada16d5f1476f9c944b24135edef493473b8b4bdda3c9d540658 Jul 24 15:45:28.502668 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:28.502442 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fd1def_d92a_438b_a6a1_0fec4b699fea.slice/crio-e21bd25d137b7c4aed2ee5c30e04ef88cd74dbdd557840029a852c7811fac23f WatchSource:0}: Error finding container e21bd25d137b7c4aed2ee5c30e04ef88cd74dbdd557840029a852c7811fac23f: Status 404 returned error can't find the container with id e21bd25d137b7c4aed2ee5c30e04ef88cd74dbdd557840029a852c7811fac23f Jul 24 15:45:28.503942 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:45:28.503728 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357d493f_b503_4d18_b37c_e8e7168f84c2.slice/crio-1aac7af750c821fe7cf8b84970e9b986d3aa6829bdfbe04b67ae22d42b331b9d WatchSource:0}: Error finding container 1aac7af750c821fe7cf8b84970e9b986d3aa6829bdfbe04b67ae22d42b331b9d: Status 404 returned error can't find the container with id 1aac7af750c821fe7cf8b84970e9b986d3aa6829bdfbe04b67ae22d42b331b9d Jul 24 15:45:28.636966 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:28.636927 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g2vqr" event={"ID":"357d493f-b503-4d18-b37c-e8e7168f84c2","Type":"ContainerStarted","Data":"1aac7af750c821fe7cf8b84970e9b986d3aa6829bdfbe04b67ae22d42b331b9d"} Jul 24 15:45:28.638044 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:28.638016 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" event={"ID":"c1fd1def-d92a-438b-a6a1-0fec4b699fea","Type":"ContainerStarted","Data":"e21bd25d137b7c4aed2ee5c30e04ef88cd74dbdd557840029a852c7811fac23f"} Jul 24 15:45:28.639188 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:28.639153 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" event={"ID":"5effb096-2376-43a2-83bd-4c3f3f085333","Type":"ContainerStarted","Data":"4baade4036db7841ff68f2892196d45f51e1f10efeaeffc8a10e919fb03284c3"} Jul 24 15:45:28.640204 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:28.640183 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" event={"ID":"fc74e7b2-f0e1-4fe3-b02a-567e2f726143","Type":"ContainerStarted","Data":"fca208f664688c00ee1141abe2f71e2b8eead5d369b30b3d12e9cc172aa04e0d"} Jul 24 15:45:28.641381 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:28.641353 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkwbb" event={"ID":"6f58d4fe-4a3c-466c-8001-85fd8797a32a","Type":"ContainerStarted","Data":"dcfde2d9411ce266e07f1e5097aa54d5489715033b7c433967fe6a295e5c3446"} Jul 24 15:45:28.643204 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:28.643183 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal" event={"ID":"7dc55cd8cdabaf6971bfe9f07f04cbd9","Type":"ContainerStarted","Data":"af932894c1a53fb1794943d31f26313eabe84029e6ba6407d51bd24b3f3df0e2"} Jul 24 15:45:28.644260 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:28.644235 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bdjkj" event={"ID":"3447e518-ab20-411f-aa9f-4918b9246385","Type":"ContainerStarted","Data":"4443843202d3ada16d5f1476f9c944b24135edef493473b8b4bdda3c9d540658"} Jul 24 15:45:28.645419 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:28.645397 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-txrqw" event={"ID":"0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8","Type":"ContainerStarted","Data":"3e74b988683e2c8e3b90671262bd9c3a5ed39b65f9620d3ce987ebafda19d47c"} Jul 24 15:45:28.646585 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:28.646566 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hvtww" event={"ID":"34e569a7-61a3-4909-865f-36a7bd286efa","Type":"ContainerStarted","Data":"0434834c2454e5501cfe021a598ae3fb8935ab55d0016f3549b74d2c6923a136"} Jul 24 15:45:28.647558 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:28.647533 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d47kt" event={"ID":"43d9054e-c012-47be-8143-968de45e5b9b","Type":"ContainerStarted","Data":"6e6da93ff9c3de2e44cd727261b3b7d9c2cbf570f508d06fe2dfee3f2fbffa4b"} Jul 24 15:45:28.660048 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:28.659993 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-58.ec2.internal" podStartSLOduration=2.659975807 podStartE2EDuration="2.659975807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:28.659574071 +0000 UTC m=+3.577544371" watchObservedRunningTime="2026-07-24 15:45:28.659975807 +0000 UTC m=+3.577946108" Jul 24 15:45:29.042157 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:29.042070 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs\") pod \"network-metrics-daemon-pdlxm\" (UID: \"2ff63e8f-2348-417f-a83d-8e72db071ae7\") " pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:29.042157 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:29.042153 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret\") pod \"global-pull-secret-syncer-lj6pt\" (UID: \"d1348e8a-07ee-4ce2-afa4-2d580edd8df2\") " pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:29.042397 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:29.042322 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:45:29.042397 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:29.042385 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret podName:d1348e8a-07ee-4ce2-afa4-2d580edd8df2 nodeName:}" failed. No retries permitted until 2026-07-24 15:45:31.042366033 +0000 UTC m=+5.960336333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret") pod "global-pull-secret-syncer-lj6pt" (UID: "d1348e8a-07ee-4ce2-afa4-2d580edd8df2") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:45:29.042840 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:29.042818 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:45:29.042927 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:29.042881 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs podName:2ff63e8f-2348-417f-a83d-8e72db071ae7 nodeName:}" failed. No retries permitted until 2026-07-24 15:45:31.04286446 +0000 UTC m=+5.960834754 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs") pod "network-metrics-daemon-pdlxm" (UID: "2ff63e8f-2348-417f-a83d-8e72db071ae7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:45:29.143560 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:29.143250 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd7f5\" (UniqueName: \"kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5\") pod \"network-check-target-tchwh\" (UID: \"2615ac1e-bb0a-4653-a612-f1406ea7d173\") " pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:29.143560 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:29.143432 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:45:29.143560 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:29.143452 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:45:29.143560 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:29.143464 2569 projected.go:196] Error preparing data for projected volume kube-api-access-bd7f5 for pod openshift-network-diagnostics/network-check-target-tchwh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:45:29.143560 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:29.143523 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5 podName:2615ac1e-bb0a-4653-a612-f1406ea7d173 nodeName:}" failed. No retries permitted until 2026-07-24 15:45:31.143504995 +0000 UTC m=+6.061475274 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bd7f5" (UniqueName: "kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5") pod "network-check-target-tchwh" (UID: "2615ac1e-bb0a-4653-a612-f1406ea7d173") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:45:29.480308 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:29.480154 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 15:40:27 +0000 UTC" deadline="2028-03-21 01:49:50.268662124 +0000 UTC" Jul 24 15:45:29.480308 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:29.480193 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14530h4m20.788473376s" Jul 24 15:45:29.622416 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:29.621559 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:29.622416 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:29.621696 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lj6pt" podUID="d1348e8a-07ee-4ce2-afa4-2d580edd8df2" Jul 24 15:45:29.622416 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:29.622181 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:29.622416 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:29.622277 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tchwh" podUID="2615ac1e-bb0a-4653-a612-f1406ea7d173" Jul 24 15:45:29.622416 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:29.622335 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:29.622416 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:29.622386 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlxm" podUID="2ff63e8f-2348-417f-a83d-8e72db071ae7" Jul 24 15:45:29.670817 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:29.670782 2569 generic.go:356] "Generic (PLEG): container finished" podID="f14b58b36cb1a9f322a5dc26c775590d" containerID="684948226d00de2f0d1e81a0a329c62eb8a95c33a2d74321f0227d53f2869eb9" exitCode=0 Jul 24 15:45:29.671929 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:29.671902 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal" event={"ID":"f14b58b36cb1a9f322a5dc26c775590d","Type":"ContainerDied","Data":"684948226d00de2f0d1e81a0a329c62eb8a95c33a2d74321f0227d53f2869eb9"} Jul 24 15:45:30.697026 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:30.696892 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal" event={"ID":"f14b58b36cb1a9f322a5dc26c775590d","Type":"ContainerStarted","Data":"15a136aaf0db4e29b7e2dcce1b33a0968e0ac1aa0a29e1bb0d3ac03ac59edd02"} Jul 24 15:45:31.066867 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:31.066778 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs\") pod \"network-metrics-daemon-pdlxm\" (UID: \"2ff63e8f-2348-417f-a83d-8e72db071ae7\") " pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:31.066867 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:31.066848 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret\") pod \"global-pull-secret-syncer-lj6pt\" (UID: \"d1348e8a-07ee-4ce2-afa4-2d580edd8df2\") " pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:31.067087 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:31.066987 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:45:31.067087 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:31.067049 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret podName:d1348e8a-07ee-4ce2-afa4-2d580edd8df2 nodeName:}" failed. No retries permitted until 2026-07-24 15:45:35.067030964 +0000 UTC m=+9.985001245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret") pod "global-pull-secret-syncer-lj6pt" (UID: "d1348e8a-07ee-4ce2-afa4-2d580edd8df2") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:45:31.067538 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:31.067488 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:45:31.067632 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:31.067543 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs podName:2ff63e8f-2348-417f-a83d-8e72db071ae7 nodeName:}" failed. No retries permitted until 2026-07-24 15:45:35.067527685 +0000 UTC m=+9.985497967 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs") pod "network-metrics-daemon-pdlxm" (UID: "2ff63e8f-2348-417f-a83d-8e72db071ae7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:45:31.168327 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:31.168289 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd7f5\" (UniqueName: \"kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5\") pod \"network-check-target-tchwh\" (UID: \"2615ac1e-bb0a-4653-a612-f1406ea7d173\") " pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:31.168522 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:31.168472 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:45:31.168522 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:31.168494 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:45:31.168522 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:31.168507 2569 projected.go:196] Error preparing data for projected volume kube-api-access-bd7f5 for pod openshift-network-diagnostics/network-check-target-tchwh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:45:31.168682 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:31.168596 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5 podName:2615ac1e-bb0a-4653-a612-f1406ea7d173 nodeName:}" failed. No retries permitted until 2026-07-24 15:45:35.168575817 +0000 UTC m=+10.086546097 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bd7f5" (UniqueName: "kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5") pod "network-check-target-tchwh" (UID: "2615ac1e-bb0a-4653-a612-f1406ea7d173") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:45:31.625116 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:31.624271 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:31.625116 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:31.624405 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lj6pt" podUID="d1348e8a-07ee-4ce2-afa4-2d580edd8df2" Jul 24 15:45:31.625116 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:31.624835 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:31.625116 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:31.624924 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tchwh" podUID="2615ac1e-bb0a-4653-a612-f1406ea7d173" Jul 24 15:45:31.625116 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:31.624981 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:31.625116 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:31.625066 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlxm" podUID="2ff63e8f-2348-417f-a83d-8e72db071ae7" Jul 24 15:45:33.621806 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:33.621761 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:33.622281 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:33.621761 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:33.622281 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:33.621908 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tchwh" podUID="2615ac1e-bb0a-4653-a612-f1406ea7d173" Jul 24 15:45:33.622281 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:33.621990 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lj6pt" podUID="d1348e8a-07ee-4ce2-afa4-2d580edd8df2" Jul 24 15:45:33.622281 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:33.621771 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:33.622281 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:33.622093 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlxm" podUID="2ff63e8f-2348-417f-a83d-8e72db071ae7" Jul 24 15:45:35.105183 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:35.104547 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret\") pod \"global-pull-secret-syncer-lj6pt\" (UID: \"d1348e8a-07ee-4ce2-afa4-2d580edd8df2\") " pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:35.105183 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:35.104636 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs\") pod \"network-metrics-daemon-pdlxm\" (UID: \"2ff63e8f-2348-417f-a83d-8e72db071ae7\") " pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:35.105183 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:35.104763 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:45:35.105183 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:35.104825 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs podName:2ff63e8f-2348-417f-a83d-8e72db071ae7 nodeName:}" failed. No retries permitted until 2026-07-24 15:45:43.104806083 +0000 UTC m=+18.022776375 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs") pod "network-metrics-daemon-pdlxm" (UID: "2ff63e8f-2348-417f-a83d-8e72db071ae7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:45:35.105183 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:35.105109 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:45:35.105183 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:35.105156 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret podName:d1348e8a-07ee-4ce2-afa4-2d580edd8df2 nodeName:}" failed. No retries permitted until 2026-07-24 15:45:43.105141735 +0000 UTC m=+18.023112028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret") pod "global-pull-secret-syncer-lj6pt" (UID: "d1348e8a-07ee-4ce2-afa4-2d580edd8df2") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:45:35.205530 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:35.205491 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd7f5\" (UniqueName: \"kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5\") pod \"network-check-target-tchwh\" (UID: \"2615ac1e-bb0a-4653-a612-f1406ea7d173\") " pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:35.205718 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:35.205706 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:45:35.205783 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:35.205728 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:45:35.205783 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:35.205740 2569 projected.go:196] Error preparing data for projected volume kube-api-access-bd7f5 for pod openshift-network-diagnostics/network-check-target-tchwh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:45:35.205887 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:35.205805 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5 podName:2615ac1e-bb0a-4653-a612-f1406ea7d173 nodeName:}" failed. No retries permitted until 2026-07-24 15:45:43.205788086 +0000 UTC m=+18.123758367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bd7f5" (UniqueName: "kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5") pod "network-check-target-tchwh" (UID: "2615ac1e-bb0a-4653-a612-f1406ea7d173") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:45:35.621843 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:35.621811 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:35.622023 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:35.621823 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:35.623441 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:35.623073 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tchwh" podUID="2615ac1e-bb0a-4653-a612-f1406ea7d173" Jul 24 15:45:35.623441 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:35.623119 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:35.623441 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:35.623181 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lj6pt" podUID="d1348e8a-07ee-4ce2-afa4-2d580edd8df2" Jul 24 15:45:35.623441 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:35.623187 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlxm" podUID="2ff63e8f-2348-417f-a83d-8e72db071ae7" Jul 24 15:45:37.623241 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:37.622295 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:37.623241 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:37.622434 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lj6pt" podUID="d1348e8a-07ee-4ce2-afa4-2d580edd8df2" Jul 24 15:45:37.623241 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:37.622828 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:37.623241 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:37.622970 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tchwh" podUID="2615ac1e-bb0a-4653-a612-f1406ea7d173" Jul 24 15:45:37.623241 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:37.623109 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:37.623241 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:37.623203 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlxm" podUID="2ff63e8f-2348-417f-a83d-8e72db071ae7" Jul 24 15:45:39.624202 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:39.624122 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:39.624202 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:39.624155 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:39.624697 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:39.624125 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:39.624697 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:39.624267 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lj6pt" podUID="d1348e8a-07ee-4ce2-afa4-2d580edd8df2" Jul 24 15:45:39.624697 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:39.624349 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlxm" podUID="2ff63e8f-2348-417f-a83d-8e72db071ae7" Jul 24 15:45:39.624697 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:39.624439 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tchwh" podUID="2615ac1e-bb0a-4653-a612-f1406ea7d173" Jul 24 15:45:41.621982 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:41.621950 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:41.622419 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:41.621950 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:41.622419 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:41.622068 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlxm" podUID="2ff63e8f-2348-417f-a83d-8e72db071ae7" Jul 24 15:45:41.622419 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:41.622174 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tchwh" podUID="2615ac1e-bb0a-4653-a612-f1406ea7d173" Jul 24 15:45:41.622419 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:41.622180 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:41.622419 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:41.622274 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lj6pt" podUID="d1348e8a-07ee-4ce2-afa4-2d580edd8df2" Jul 24 15:45:43.167580 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:43.167473 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret\") pod \"global-pull-secret-syncer-lj6pt\" (UID: \"d1348e8a-07ee-4ce2-afa4-2d580edd8df2\") " pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:43.167580 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:43.167572 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs\") pod \"network-metrics-daemon-pdlxm\" (UID: \"2ff63e8f-2348-417f-a83d-8e72db071ae7\") " pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:43.168125 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:43.167650 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:45:43.168125 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:43.167699 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:45:43.168125 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:43.167725 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret podName:d1348e8a-07ee-4ce2-afa4-2d580edd8df2 nodeName:}" failed. No retries permitted until 2026-07-24 15:45:59.167706173 +0000 UTC m=+34.085676455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret") pod "global-pull-secret-syncer-lj6pt" (UID: "d1348e8a-07ee-4ce2-afa4-2d580edd8df2") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:45:43.168125 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:43.167769 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs podName:2ff63e8f-2348-417f-a83d-8e72db071ae7 nodeName:}" failed. No retries permitted until 2026-07-24 15:45:59.167748564 +0000 UTC m=+34.085718895 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs") pod "network-metrics-daemon-pdlxm" (UID: "2ff63e8f-2348-417f-a83d-8e72db071ae7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:45:43.268720 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:43.268680 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd7f5\" (UniqueName: \"kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5\") pod \"network-check-target-tchwh\" (UID: \"2615ac1e-bb0a-4653-a612-f1406ea7d173\") " pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:43.268893 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:43.268860 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:45:43.268893 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:43.268888 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:45:43.268988 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:43.268901 2569 projected.go:196] Error preparing data for projected volume kube-api-access-bd7f5 for pod openshift-network-diagnostics/network-check-target-tchwh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:45:43.268988 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:43.268972 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5 podName:2615ac1e-bb0a-4653-a612-f1406ea7d173 nodeName:}" failed. No retries permitted until 2026-07-24 15:45:59.268951313 +0000 UTC m=+34.186921618 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bd7f5" (UniqueName: "kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5") pod "network-check-target-tchwh" (UID: "2615ac1e-bb0a-4653-a612-f1406ea7d173") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:45:43.621577 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:43.621483 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:43.621729 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:43.621496 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:43.621729 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:43.621612 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lj6pt" podUID="d1348e8a-07ee-4ce2-afa4-2d580edd8df2" Jul 24 15:45:43.621729 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:43.621496 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:43.621729 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:43.621695 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlxm" podUID="2ff63e8f-2348-417f-a83d-8e72db071ae7" Jul 24 15:45:43.621941 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:43.621782 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tchwh" podUID="2615ac1e-bb0a-4653-a612-f1406ea7d173" Jul 24 15:45:45.621960 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:45.621928 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:45.622380 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:45.622017 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tchwh" podUID="2615ac1e-bb0a-4653-a612-f1406ea7d173" Jul 24 15:45:45.622380 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:45.622131 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:45.622380 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:45.622262 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlxm" podUID="2ff63e8f-2348-417f-a83d-8e72db071ae7" Jul 24 15:45:45.622380 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:45.622310 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:45.622540 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:45.622382 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lj6pt" podUID="d1348e8a-07ee-4ce2-afa4-2d580edd8df2" Jul 24 15:45:47.622002 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.621823 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:47.622860 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.621829 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:47.622860 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:47.622080 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lj6pt" podUID="d1348e8a-07ee-4ce2-afa4-2d580edd8df2" Jul 24 15:45:47.622860 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.621867 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:47.622860 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:47.622156 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tchwh" podUID="2615ac1e-bb0a-4653-a612-f1406ea7d173" Jul 24 15:45:47.622860 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:47.622254 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlxm" podUID="2ff63e8f-2348-417f-a83d-8e72db071ae7" Jul 24 15:45:47.732996 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.732971 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d47kt" event={"ID":"43d9054e-c012-47be-8143-968de45e5b9b","Type":"ContainerStarted","Data":"abc128b5e4b7912e6976d33626d43d95e0191893cd7ce7bc8caf51c2b9664e9f"} Jul 24 15:45:47.734341 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.734314 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g2vqr" event={"ID":"357d493f-b503-4d18-b37c-e8e7168f84c2","Type":"ContainerStarted","Data":"e04b8de6a530f2e4720b968478361cec7aa2ae2473e44377140858fa296e3e68"} Jul 24 15:45:47.735587 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.735566 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" event={"ID":"c1fd1def-d92a-438b-a6a1-0fec4b699fea","Type":"ContainerStarted","Data":"23f7324e4c129a278bb5bd6b9ab3a2ed2f3e4482680ac2482e7319e89cded832"} Jul 24 15:45:47.736883 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.736863 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" event={"ID":"5effb096-2376-43a2-83bd-4c3f3f085333","Type":"ContainerStarted","Data":"8ac73a6a91941c91fc53b4d986b8481742c9ed581e3797d6f41eb3daaaf126fc"} Jul 24 15:45:47.738594 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.738579 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 15:45:47.738890 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.738874 2569 generic.go:356] "Generic (PLEG): container finished" podID="fc74e7b2-f0e1-4fe3-b02a-567e2f726143" containerID="be051657525c55c7d8dbc83264d8ef7709295c4b7b6e1ccf6f5ac8b72feccb61" exitCode=1 Jul 24 15:45:47.738938 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.738930 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" event={"ID":"fc74e7b2-f0e1-4fe3-b02a-567e2f726143","Type":"ContainerStarted","Data":"c3809045e9c42d2c1f0a90f81d37463e2ef630e62029c8863f448bec529f99ec"} Jul 24 15:45:47.738989 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.738977 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" event={"ID":"fc74e7b2-f0e1-4fe3-b02a-567e2f726143","Type":"ContainerStarted","Data":"6db88f72b405bbecdef8cc16a68822be076af23a51d916cb7819837022e56fa8"} Jul 24 15:45:47.739035 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.739028 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" event={"ID":"fc74e7b2-f0e1-4fe3-b02a-567e2f726143","Type":"ContainerDied","Data":"be051657525c55c7d8dbc83264d8ef7709295c4b7b6e1ccf6f5ac8b72feccb61"} Jul 24 15:45:47.739070 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.739058 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" event={"ID":"fc74e7b2-f0e1-4fe3-b02a-567e2f726143","Type":"ContainerStarted","Data":"3b99d3d1fda6a38cfafed9d94ec3dfd5ef02d5e865e1f638ae5ea02874e62a78"} Jul 24 15:45:47.740016 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.739999 2569 generic.go:356] "Generic (PLEG): container finished" podID="6f58d4fe-4a3c-466c-8001-85fd8797a32a" containerID="dcac2dd9a04282654ce3e391d32590e8ff9c0612de6c561eafb987bd899d3ee0" exitCode=0 Jul 24 15:45:47.740086 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.740064 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkwbb" event={"ID":"6f58d4fe-4a3c-466c-8001-85fd8797a32a","Type":"ContainerDied","Data":"dcac2dd9a04282654ce3e391d32590e8ff9c0612de6c561eafb987bd899d3ee0"} Jul 24 15:45:47.744256 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.744231 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bdjkj" event={"ID":"3447e518-ab20-411f-aa9f-4918b9246385","Type":"ContainerStarted","Data":"6b25ea28a213443b3f71331e8d025376eed5c2e22117c74d20fd3ec73611d00f"} Jul 24 15:45:47.745682 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.745661 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hvtww" event={"ID":"34e569a7-61a3-4909-865f-36a7bd286efa","Type":"ContainerStarted","Data":"559f0c0f13f51ac739026fb1dab7b3f53eefee4dcb139497bbee95593b38d73f"} Jul 24 15:45:47.749563 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.749467 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-58.ec2.internal" podStartSLOduration=21.749455276 podStartE2EDuration="21.749455276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:30.714283634 +0000 UTC m=+5.632253939" watchObservedRunningTime="2026-07-24 15:45:47.749455276 +0000 UTC m=+22.667425597" Jul 24 15:45:47.749761 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.749741 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-d47kt" podStartSLOduration=4.077028087 podStartE2EDuration="22.749735921s" totalImagesPullingTime="18.672707834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:47.749278816 +0000 UTC m=+22.667249120" watchObservedRunningTime="2026-07-24 15:45:47.749735921 +0000 UTC m=+22.667706222" Jul 24 15:45:47.766377 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.766341 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bdjkj" podStartSLOduration=4.121502849 podStartE2EDuration="22.766329625s" totalImagesPullingTime="18.644826776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:47.766085514 +0000 UTC m=+22.684055813" watchObservedRunningTime="2026-07-24 15:45:47.766329625 +0000 UTC m=+22.684299914" Jul 24 15:45:47.779522 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.779489 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g2vqr" podStartSLOduration=12.099405673 podStartE2EDuration="22.779478599s" totalImagesPullingTime="10.680072926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:47.779200816 +0000 UTC m=+22.697171113" watchObservedRunningTime="2026-07-24 15:45:47.779478599 +0000 UTC m=+22.697448898" Jul 24 15:45:47.801114 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.801061 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6ztbq" podStartSLOduration=4.136300888 podStartE2EDuration="22.801042582s" totalImagesPullingTime="18.664741694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:47.801030861 +0000 UTC m=+22.719001162" watchObservedRunningTime="2026-07-24 15:45:47.801042582 +0000 UTC m=+22.719012882" Jul 24 15:45:47.838810 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:47.838764 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hvtww" podStartSLOduration=4.189201133 podStartE2EDuration="22.838749741s" totalImagesPullingTime="18.649548608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:47.838641205 +0000 UTC m=+22.756611503" watchObservedRunningTime="2026-07-24 15:45:47.838749741 +0000 UTC m=+22.756720040" Jul 24 15:45:48.749396 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:48.749309 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-txrqw" event={"ID":"0a5f0ba5-21bf-4de8-9b8b-3f09ad6e5bc8","Type":"ContainerStarted","Data":"10d4436b34a8676f179e7377671cc5f7d39e230717fdb50952b2b9b2ff476b13"} Jul 24 15:45:48.752912 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:48.752890 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 15:45:48.753265 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:48.753240 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" event={"ID":"fc74e7b2-f0e1-4fe3-b02a-567e2f726143","Type":"ContainerStarted","Data":"168d0245f1b8307c10fde18a0c9ac37361371587bc066888be6a343a519b4ffc"} Jul 24 15:45:48.753344 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:48.753315 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" event={"ID":"fc74e7b2-f0e1-4fe3-b02a-567e2f726143","Type":"ContainerStarted","Data":"224d7bcc561793e71408720d50704a053e7a743407a579cb04d9604d341b27d5"} Jul 24 15:45:48.767453 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:48.767406 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-txrqw" podStartSLOduration=5.121553136 podStartE2EDuration="23.767390764s" totalImagesPullingTime="18.645837628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:48.766835596 +0000 UTC m=+23.684805896" watchObservedRunningTime="2026-07-24 15:45:48.767390764 +0000 UTC m=+23.685361064" Jul 24 15:45:49.265809 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:49.265786 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 15:45:49.491116 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:49.491006 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T15:45:49.265804092Z","UUID":"138ad805-7f81-45ce-8d6b-8a5e686ec4e8","Handler":null,"Name":"","Endpoint":""} Jul 24 15:45:49.492801 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:49.492766 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 15:45:49.492934 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:49.492814 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 15:45:49.621088 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:49.621059 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:49.621339 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:49.621067 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:49.621339 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:49.621225 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:49.621339 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:49.621177 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tchwh" podUID="2615ac1e-bb0a-4653-a612-f1406ea7d173" Jul 24 15:45:49.621339 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:49.621320 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlxm" podUID="2ff63e8f-2348-417f-a83d-8e72db071ae7" Jul 24 15:45:49.621557 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:49.621418 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lj6pt" podUID="d1348e8a-07ee-4ce2-afa4-2d580edd8df2" Jul 24 15:45:49.645286 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:49.645260 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g2vqr" Jul 24 15:45:49.645812 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:49.645799 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g2vqr" Jul 24 15:45:49.757381 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:49.757294 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" event={"ID":"5effb096-2376-43a2-83bd-4c3f3f085333","Type":"ContainerStarted","Data":"e2dd822ae76131709402ef483935501a9376009e230e887cf351433f80826113"} Jul 24 15:45:49.757886 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:49.757655 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g2vqr" Jul 24 15:45:49.758131 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:49.758118 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g2vqr" Jul 24 15:45:50.763750 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:50.763722 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 15:45:50.764334 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:50.764136 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" event={"ID":"fc74e7b2-f0e1-4fe3-b02a-567e2f726143","Type":"ContainerStarted","Data":"ef217def0d4c7bed23ca016e2ae5367ebc99523b4c31c92838389dc1be6d225d"} Jul 24 15:45:51.621088 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:51.621050 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:51.621088 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:51.621075 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:51.621358 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:51.621048 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:51.621358 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:51.621178 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lj6pt" podUID="d1348e8a-07ee-4ce2-afa4-2d580edd8df2" Jul 24 15:45:51.621358 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:51.621336 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tchwh" podUID="2615ac1e-bb0a-4653-a612-f1406ea7d173" Jul 24 15:45:51.621521 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:51.621457 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlxm" podUID="2ff63e8f-2348-417f-a83d-8e72db071ae7" Jul 24 15:45:52.770812 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:52.770585 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" event={"ID":"5effb096-2376-43a2-83bd-4c3f3f085333","Type":"ContainerStarted","Data":"262b7c7c7b95ea22b4c70c8eea285adef65580438d3eefac0ae960b4aa4a4c71"} Jul 24 15:45:52.773988 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:52.773967 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 15:45:52.774383 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:52.774351 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" event={"ID":"fc74e7b2-f0e1-4fe3-b02a-567e2f726143","Type":"ContainerStarted","Data":"7461eb761bc39004e036372c94d4b24f331dd3da9d0839ce14fe9e7e6a231021"} Jul 24 15:45:52.774599 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:52.774583 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:52.774757 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:52.774741 2569 scope.go:117] "RemoveContainer" containerID="be051657525c55c7d8dbc83264d8ef7709295c4b7b6e1ccf6f5ac8b72feccb61" Jul 24 15:45:52.776276 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:52.776252 2569 generic.go:356] "Generic (PLEG): container finished" podID="6f58d4fe-4a3c-466c-8001-85fd8797a32a" containerID="c8616beec50e70555b149cbd048a93361028b279a86e42b4640ac4f53457d20b" exitCode=0 Jul 24 15:45:52.776354 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:52.776286 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkwbb" event={"ID":"6f58d4fe-4a3c-466c-8001-85fd8797a32a","Type":"ContainerDied","Data":"c8616beec50e70555b149cbd048a93361028b279a86e42b4640ac4f53457d20b"} Jul 24 15:45:52.791065 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:52.791043 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:52.791188 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:52.791146 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rlbpx" podStartSLOduration=4.485370735 podStartE2EDuration="27.791133272s" totalImagesPullingTime="23.305762537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:45:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:52.790577259 +0000 UTC m=+27.708547561" watchObservedRunningTime="2026-07-24 15:45:52.791133272 +0000 UTC m=+27.709103572" Jul 24 15:45:53.621616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:53.621583 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:53.621825 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:53.621583 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:53.621825 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:53.621713 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlxm" podUID="2ff63e8f-2348-417f-a83d-8e72db071ae7" Jul 24 15:45:53.621825 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:53.621583 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:53.621825 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:53.621770 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tchwh" podUID="2615ac1e-bb0a-4653-a612-f1406ea7d173" Jul 24 15:45:53.622000 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:53.621877 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lj6pt" podUID="d1348e8a-07ee-4ce2-afa4-2d580edd8df2" Jul 24 15:45:53.782078 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:53.782012 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 15:45:53.782518 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:53.782380 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" event={"ID":"fc74e7b2-f0e1-4fe3-b02a-567e2f726143","Type":"ContainerStarted","Data":"bfceb03cf07ba4a4b88cbd345a7bb6ad77a5a8d17325b571d0705d6890bd0058"} Jul 24 15:45:53.782659 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:53.782641 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:53.782718 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:53.782696 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:53.798735 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:53.798710 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:45:53.813021 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:53.812971 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" podStartSLOduration=9.983899591 podStartE2EDuration="28.812957279s" totalImagesPullingTime="18.829057688s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:53.812831868 +0000 UTC m=+28.730802170" watchObservedRunningTime="2026-07-24 15:45:53.812957279 +0000 UTC m=+28.730927579" Jul 24 15:45:54.436579 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:54.436321 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lj6pt"] Jul 24 15:45:54.436772 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:54.436597 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:54.436772 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:54.436685 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lj6pt" podUID="d1348e8a-07ee-4ce2-afa4-2d580edd8df2" Jul 24 15:45:54.440289 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:54.440258 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tchwh"] Jul 24 15:45:54.440447 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:54.440378 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:54.440523 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:54.440475 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tchwh" podUID="2615ac1e-bb0a-4653-a612-f1406ea7d173" Jul 24 15:45:54.440892 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:54.440873 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pdlxm"] Jul 24 15:45:54.440974 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:54.440962 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:54.441062 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:54.441043 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlxm" podUID="2ff63e8f-2348-417f-a83d-8e72db071ae7" Jul 24 15:45:54.787220 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:54.787121 2569 generic.go:356] "Generic (PLEG): container finished" podID="6f58d4fe-4a3c-466c-8001-85fd8797a32a" containerID="115813f3a53dc80bb17dde6ea5c8521e7a0c613376b7a51b74cbbdd6f11a6faa" exitCode=0 Jul 24 15:45:54.787555 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:54.787207 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkwbb" event={"ID":"6f58d4fe-4a3c-466c-8001-85fd8797a32a","Type":"ContainerDied","Data":"115813f3a53dc80bb17dde6ea5c8521e7a0c613376b7a51b74cbbdd6f11a6faa"} Jul 24 15:45:55.622409 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:55.622376 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:55.622715 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:55.622494 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lj6pt" podUID="d1348e8a-07ee-4ce2-afa4-2d580edd8df2" Jul 24 15:45:55.792303 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:55.792271 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkwbb" event={"ID":"6f58d4fe-4a3c-466c-8001-85fd8797a32a","Type":"ContainerStarted","Data":"3aeb21232877b2c38a5e15cd8c8630802fcc2d033236c11b153377b8ee1daeec"} Jul 24 15:45:56.621773 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:56.621694 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:56.621773 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:56.621724 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:56.621941 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:56.621849 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tchwh" podUID="2615ac1e-bb0a-4653-a612-f1406ea7d173" Jul 24 15:45:56.621975 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:56.621957 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlxm" podUID="2ff63e8f-2348-417f-a83d-8e72db071ae7" Jul 24 15:45:56.797092 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:56.797059 2569 generic.go:356] "Generic (PLEG): container finished" podID="6f58d4fe-4a3c-466c-8001-85fd8797a32a" containerID="3aeb21232877b2c38a5e15cd8c8630802fcc2d033236c11b153377b8ee1daeec" exitCode=0 Jul 24 15:45:56.797598 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:56.797112 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkwbb" event={"ID":"6f58d4fe-4a3c-466c-8001-85fd8797a32a","Type":"ContainerDied","Data":"3aeb21232877b2c38a5e15cd8c8630802fcc2d033236c11b153377b8ee1daeec"} Jul 24 15:45:57.622110 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:57.622076 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:57.622297 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:57.622207 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lj6pt" podUID="d1348e8a-07ee-4ce2-afa4-2d580edd8df2" Jul 24 15:45:58.621591 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:58.621559 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:58.622164 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:58.621693 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlxm" podUID="2ff63e8f-2348-417f-a83d-8e72db071ae7" Jul 24 15:45:58.622164 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:58.621735 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:58.622164 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:58.621810 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tchwh" podUID="2615ac1e-bb0a-4653-a612-f1406ea7d173" Jul 24 15:45:59.192243 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:59.192179 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret\") pod \"global-pull-secret-syncer-lj6pt\" (UID: \"d1348e8a-07ee-4ce2-afa4-2d580edd8df2\") " pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:59.192450 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:59.192301 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs\") pod \"network-metrics-daemon-pdlxm\" (UID: \"2ff63e8f-2348-417f-a83d-8e72db071ae7\") " pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:45:59.192450 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:59.192352 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:45:59.192450 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:59.192420 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:45:59.192450 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:59.192432 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret podName:d1348e8a-07ee-4ce2-afa4-2d580edd8df2 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:31.192411936 +0000 UTC m=+66.110382234 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret") pod "global-pull-secret-syncer-lj6pt" (UID: "d1348e8a-07ee-4ce2-afa4-2d580edd8df2") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:45:59.192652 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:59.192462 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs podName:2ff63e8f-2348-417f-a83d-8e72db071ae7 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:31.192450963 +0000 UTC m=+66.110421242 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs") pod "network-metrics-daemon-pdlxm" (UID: "2ff63e8f-2348-417f-a83d-8e72db071ae7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:45:59.292834 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:59.292801 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd7f5\" (UniqueName: \"kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5\") pod \"network-check-target-tchwh\" (UID: \"2615ac1e-bb0a-4653-a612-f1406ea7d173\") " pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:45:59.293033 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:59.292936 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:45:59.293033 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:59.292954 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:45:59.293033 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:59.292963 2569 projected.go:196] Error preparing data for projected volume kube-api-access-bd7f5 for pod openshift-network-diagnostics/network-check-target-tchwh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:45:59.293033 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:59.293025 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5 podName:2615ac1e-bb0a-4653-a612-f1406ea7d173 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:31.293005847 +0000 UTC m=+66.210976145 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bd7f5" (UniqueName: "kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5") pod "network-check-target-tchwh" (UID: "2615ac1e-bb0a-4653-a612-f1406ea7d173") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:45:59.621517 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:45:59.621428 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:45:59.621671 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:45:59.621570 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lj6pt" podUID="d1348e8a-07ee-4ce2-afa4-2d580edd8df2" Jul 24 15:46:00.621872 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.621837 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:46:00.622313 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.621880 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:46:00.622313 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:00.621979 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pdlxm" podUID="2ff63e8f-2348-417f-a83d-8e72db071ae7" Jul 24 15:46:00.622313 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:00.622046 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tchwh" podUID="2615ac1e-bb0a-4653-a612-f1406ea7d173" Jul 24 15:46:00.891422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.891250 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-58.ec2.internal" event="NodeReady" Jul 24 15:46:00.891573 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.891531 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 15:46:00.926744 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.926710 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d75669b64-9vf6v"] Jul 24 15:46:00.947980 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.947948 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d8p9n"] Jul 24 15:46:00.948237 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.948088 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:00.950884 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.950858 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 24 15:46:00.951277 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.951256 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nsfqx\"" Jul 24 15:46:00.951541 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.951524 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 24 15:46:00.952000 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.951803 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 24 15:46:00.963596 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.963405 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 24 15:46:00.964054 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.964013 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz"] Jul 24 15:46:00.964241 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.964224 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d8p9n" Jul 24 15:46:00.968093 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.967087 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:46:00.968093 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.967361 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 24 15:46:00.968093 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.967374 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9vw84\"" Jul 24 15:46:00.979024 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.979000 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wvx4n"] Jul 24 15:46:00.979239 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.979204 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" Jul 24 15:46:00.982683 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.982661 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 24 15:46:00.982857 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.982837 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 15:46:00.982928 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.982885 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 15:46:00.982928 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.982905 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ch4fh\"" Jul 24 15:46:00.983198 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:00.983179 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 24 15:46:01.001038 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.001015 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m"] Jul 24 15:46:01.001190 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.001148 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wvx4n" Jul 24 15:46:01.003980 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.003955 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-tkbsl\"" Jul 24 15:46:01.003980 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.003977 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 24 15:46:01.004143 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.003998 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 24 15:46:01.004143 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.004004 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 24 15:46:01.004383 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.004364 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:46:01.023048 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.023017 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc"] Jul 24 15:46:01.023174 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.023137 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m" Jul 24 15:46:01.026068 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.026047 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 24 15:46:01.026268 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.026135 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 24 15:46:01.026510 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.026489 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 24 15:46:01.026832 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.026514 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:46:01.026832 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.026763 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-98d2d\"" Jul 24 15:46:01.037421 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.037404 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-d44kg"] Jul 24 15:46:01.037526 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.037511 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc" Jul 24 15:46:01.040111 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.040096 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xgk24\"" Jul 24 15:46:01.040175 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.040098 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 24 15:46:01.040400 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.040385 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:46:01.040479 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.040389 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 24 15:46:01.053991 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.053972 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-zf2c2"] Jul 24 15:46:01.054102 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.054085 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-d44kg" Jul 24 15:46:01.057528 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.057489 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 24 15:46:01.057528 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.057514 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 24 15:46:01.057689 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.057552 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-5ncdb\"" Jul 24 15:46:01.076010 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.075990 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-285sz"] Jul 24 15:46:01.076131 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.076116 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.078485 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.078465 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-h7bnw\"" Jul 24 15:46:01.078584 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.078537 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 15:46:01.078672 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.078657 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 24 15:46:01.078754 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.078738 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 15:46:01.078805 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.078769 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 24 15:46:01.084805 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.084789 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 24 15:46:01.098250 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.098227 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d75669b64-9vf6v"] Jul 24 15:46:01.098250 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.098255 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d8p9n"] Jul 24 15:46:01.098365 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.098269 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wvx4n"] Jul 24 15:46:01.098365 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.098323 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:01.098365 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.098349 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz"] Jul 24 15:46:01.098365 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.098363 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc"] Jul 24 15:46:01.098492 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.098378 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-zf2c2"] Jul 24 15:46:01.098492 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.098393 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fwkmh"] Jul 24 15:46:01.100792 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.100771 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 24 15:46:01.100792 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.100781 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 24 15:46:01.100985 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.100971 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 24 15:46:01.101047 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.101032 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:46:01.101087 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.101052 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bxrls\"" Jul 24 15:46:01.105960 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.105937 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-bound-sa-token\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.106055 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.106015 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d07fcc0-3b9f-45f4-8e85-694a9ef1483f-serving-cert\") pod \"service-ca-operator-5d6c75f84f-wvx4n\" (UID: \"8d07fcc0-3b9f-45f4-8e85-694a9ef1483f\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wvx4n" Jul 24 15:46:01.106104 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.106060 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-trusted-ca\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.106158 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.106096 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jfjh\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-kube-api-access-6jfjh\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.106243 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.106192 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/61cb38d8-9f5e-495b-931b-7561758ce3a3-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-jp9sz\" (UID: \"61cb38d8-9f5e-495b-931b-7561758ce3a3\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" Jul 24 15:46:01.106287 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.106271 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-image-registry-private-configuration\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.106327 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.106311 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jp9sz\" (UID: \"61cb38d8-9f5e-495b-931b-7561758ce3a3\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" Jul 24 15:46:01.106364 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.106349 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfnkw\" (UniqueName: \"kubernetes.io/projected/8d07fcc0-3b9f-45f4-8e85-694a9ef1483f-kube-api-access-rfnkw\") pod \"service-ca-operator-5d6c75f84f-wvx4n\" (UID: \"8d07fcc0-3b9f-45f4-8e85-694a9ef1483f\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wvx4n" Jul 24 15:46:01.106397 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.106384 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-certificates\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.106438 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.106413 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8svgm\" (UniqueName: \"kubernetes.io/projected/61cb38d8-9f5e-495b-931b-7561758ce3a3-kube-api-access-8svgm\") pod \"cluster-monitoring-operator-548b489cb8-jp9sz\" (UID: \"61cb38d8-9f5e-495b-931b-7561758ce3a3\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" Jul 24 15:46:01.106495 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.106478 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.106561 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.106544 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w6nq\" (UniqueName: \"kubernetes.io/projected/c2185c30-e6e6-4623-9bcb-1a0b6baf046d-kube-api-access-2w6nq\") pod \"volume-data-source-validator-549c6d54cf-d8p9n\" (UID: \"c2185c30-e6e6-4623-9bcb-1a0b6baf046d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d8p9n" Jul 24 15:46:01.106607 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.106581 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-ca-trust-extracted\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.106661 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.106609 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d07fcc0-3b9f-45f4-8e85-694a9ef1483f-config\") pod \"service-ca-operator-5d6c75f84f-wvx4n\" (UID: \"8d07fcc0-3b9f-45f4-8e85-694a9ef1483f\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wvx4n" Jul 24 15:46:01.106714 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.106657 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-installation-pull-secrets\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.107056 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.107042 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 24 15:46:01.110532 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.110515 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6v2fh"] Jul 24 15:46:01.110644 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.110630 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:01.113118 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.113101 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vwj8g\"" Jul 24 15:46:01.113202 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.113133 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 15:46:01.113202 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.113107 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 15:46:01.128854 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.128835 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-xg7vg"] Jul 24 15:46:01.128957 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.128941 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6v2fh" Jul 24 15:46:01.131366 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.131345 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 15:46:01.131541 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.131524 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-82l8s\"" Jul 24 15:46:01.131614 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.131529 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 15:46:01.131614 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.131579 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 15:46:01.157821 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.157740 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-65b698c97-9vtdf"] Jul 24 15:46:01.157919 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.157849 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xg7vg" Jul 24 15:46:01.162363 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.162327 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 15:46:01.162482 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.162330 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 15:46:01.162482 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.162450 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-42vjc\"" Jul 24 15:46:01.182374 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.182348 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-285sz"] Jul 24 15:46:01.182483 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.182385 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-d44kg"] Jul 24 15:46:01.182483 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.182407 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m"] Jul 24 15:46:01.182483 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.182430 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fwkmh"] Jul 24 15:46:01.182483 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.182456 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6v2fh"] Jul 24 15:46:01.182483 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.182469 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-65b698c97-9vtdf"] Jul 24 15:46:01.182483 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.182480 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-xg7vg"] Jul 24 15:46:01.182656 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.182428 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:01.184996 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.184967 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 24 15:46:01.185100 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.185056 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 24 15:46:01.185100 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.185082 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 15:46:01.185100 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.185081 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 24 15:46:01.185287 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.185061 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 24 15:46:01.185287 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.185184 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xjksm\"" Jul 24 15:46:01.185287 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.185203 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 15:46:01.207627 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.207598 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czrnr\" (UniqueName: \"kubernetes.io/projected/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-kube-api-access-czrnr\") pod \"cluster-samples-operator-6748489656-5zzlc\" (UID: \"ba4a3f79-a5dc-4a5d-b891-38bc267aed63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc" Jul 24 15:46:01.207627 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.207627 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2397af02-6f7f-4fde-905b-f733e7a77c47-snapshots\") pod \"insights-operator-5885fdb759-zf2c2\" (UID: \"2397af02-6f7f-4fde-905b-f733e7a77c47\") " pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.207824 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.207646 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2397af02-6f7f-4fde-905b-f733e7a77c47-serving-cert\") pod \"insights-operator-5885fdb759-zf2c2\" (UID: \"2397af02-6f7f-4fde-905b-f733e7a77c47\") " pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.207824 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.207664 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg6l5\" (UniqueName: \"kubernetes.io/projected/2397af02-6f7f-4fde-905b-f733e7a77c47-kube-api-access-vg6l5\") pod \"insights-operator-5885fdb759-zf2c2\" (UID: \"2397af02-6f7f-4fde-905b-f733e7a77c47\") " pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.207824 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.207777 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea54c847-2195-4b03-9e9d-31efaf46de3f-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m\" (UID: \"ea54c847-2195-4b03-9e9d-31efaf46de3f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m" Jul 24 15:46:01.207824 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.207817 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfnkw\" (UniqueName: \"kubernetes.io/projected/8d07fcc0-3b9f-45f4-8e85-694a9ef1483f-kube-api-access-rfnkw\") pod \"service-ca-operator-5d6c75f84f-wvx4n\" (UID: \"8d07fcc0-3b9f-45f4-8e85-694a9ef1483f\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wvx4n" Jul 24 15:46:01.208019 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.207849 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-certificates\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.208019 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.207887 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w6nq\" (UniqueName: \"kubernetes.io/projected/c2185c30-e6e6-4623-9bcb-1a0b6baf046d-kube-api-access-2w6nq\") pod \"volume-data-source-validator-549c6d54cf-d8p9n\" (UID: \"c2185c30-e6e6-4623-9bcb-1a0b6baf046d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d8p9n" Jul 24 15:46:01.208019 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.207919 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.208019 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.207945 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch2nn\" (UniqueName: \"kubernetes.io/projected/ea54c847-2195-4b03-9e9d-31efaf46de3f-kube-api-access-ch2nn\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m\" (UID: \"ea54c847-2195-4b03-9e9d-31efaf46de3f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m" Jul 24 15:46:01.208019 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.207983 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2397af02-6f7f-4fde-905b-f733e7a77c47-tmp\") pod \"insights-operator-5885fdb759-zf2c2\" (UID: \"2397af02-6f7f-4fde-905b-f733e7a77c47\") " pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.208019 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208010 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-ca-trust-extracted\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.208330 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208038 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d07fcc0-3b9f-45f4-8e85-694a9ef1483f-config\") pod \"service-ca-operator-5d6c75f84f-wvx4n\" (UID: \"8d07fcc0-3b9f-45f4-8e85-694a9ef1483f\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wvx4n" Jul 24 15:46:01.208330 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.208073 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:46:01.208330 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208067 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-installation-pull-secrets\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.208330 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.208093 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d75669b64-9vf6v: secret "image-registry-tls" not found Jul 24 15:46:01.208330 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208113 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea54c847-2195-4b03-9e9d-31efaf46de3f-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m\" (UID: \"ea54c847-2195-4b03-9e9d-31efaf46de3f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m" Jul 24 15:46:01.208330 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208140 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70e5d390-4438-46a0-9e53-fae9badc534f-config\") pod \"console-operator-584dcb6b54-285sz\" (UID: \"70e5d390-4438-46a0-9e53-fae9badc534f\") " pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:01.208330 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.208161 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls podName:6555d7aa-ea7e-4dbf-8bb1-be3e103afc39 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:01.708141686 +0000 UTC m=+36.626111992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls") pod "image-registry-6d75669b64-9vf6v" (UID: "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39") : secret "image-registry-tls" not found Jul 24 15:46:01.208330 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208227 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-bound-sa-token\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.208330 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208258 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70e5d390-4438-46a0-9e53-fae9badc534f-trusted-ca\") pod \"console-operator-584dcb6b54-285sz\" (UID: \"70e5d390-4438-46a0-9e53-fae9badc534f\") " pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:01.208330 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208293 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d07fcc0-3b9f-45f4-8e85-694a9ef1483f-serving-cert\") pod \"service-ca-operator-5d6c75f84f-wvx4n\" (UID: \"8d07fcc0-3b9f-45f4-8e85-694a9ef1483f\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wvx4n" Jul 24 15:46:01.208796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208337 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-trusted-ca\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.208796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208349 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-ca-trust-extracted\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.208796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208363 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jfjh\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-kube-api-access-6jfjh\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.208796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208415 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/61cb38d8-9f5e-495b-931b-7561758ce3a3-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-jp9sz\" (UID: \"61cb38d8-9f5e-495b-931b-7561758ce3a3\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" Jul 24 15:46:01.208796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208446 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr5rx\" (UniqueName: \"kubernetes.io/projected/70e5d390-4438-46a0-9e53-fae9badc534f-kube-api-access-jr5rx\") pod \"console-operator-584dcb6b54-285sz\" (UID: \"70e5d390-4438-46a0-9e53-fae9badc534f\") " pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:01.208796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208479 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-image-registry-private-configuration\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.208796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208485 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-certificates\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.208796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208508 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jp9sz\" (UID: \"61cb38d8-9f5e-495b-931b-7561758ce3a3\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" Jul 24 15:46:01.208796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208535 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2397af02-6f7f-4fde-905b-f733e7a77c47-service-ca-bundle\") pod \"insights-operator-5885fdb759-zf2c2\" (UID: \"2397af02-6f7f-4fde-905b-f733e7a77c47\") " pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.208796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208563 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8svgm\" (UniqueName: \"kubernetes.io/projected/61cb38d8-9f5e-495b-931b-7561758ce3a3-kube-api-access-8svgm\") pod \"cluster-monitoring-operator-548b489cb8-jp9sz\" (UID: \"61cb38d8-9f5e-495b-931b-7561758ce3a3\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" Jul 24 15:46:01.208796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208586 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5zzlc\" (UID: \"ba4a3f79-a5dc-4a5d-b891-38bc267aed63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc" Jul 24 15:46:01.208796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208609 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2397af02-6f7f-4fde-905b-f733e7a77c47-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-zf2c2\" (UID: \"2397af02-6f7f-4fde-905b-f733e7a77c47\") " pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.208796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208637 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d44kg\" (UID: \"2f97027f-5262-48c0-acd9-61ccc7b95bbf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d44kg" Jul 24 15:46:01.208796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208669 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2f97027f-5262-48c0-acd9-61ccc7b95bbf-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-d44kg\" (UID: \"2f97027f-5262-48c0-acd9-61ccc7b95bbf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d44kg" Jul 24 15:46:01.208796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208705 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8d07fcc0-3b9f-45f4-8e85-694a9ef1483f-config\") pod \"service-ca-operator-5d6c75f84f-wvx4n\" (UID: \"8d07fcc0-3b9f-45f4-8e85-694a9ef1483f\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wvx4n" Jul 24 15:46:01.209529 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.208710 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70e5d390-4438-46a0-9e53-fae9badc534f-serving-cert\") pod \"console-operator-584dcb6b54-285sz\" (UID: \"70e5d390-4438-46a0-9e53-fae9badc534f\") " pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:01.209529 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.208859 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 15:46:01.209529 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.208926 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls podName:61cb38d8-9f5e-495b-931b-7561758ce3a3 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:01.708906565 +0000 UTC m=+36.626876851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jp9sz" (UID: "61cb38d8-9f5e-495b-931b-7561758ce3a3") : secret "cluster-monitoring-operator-tls" not found Jul 24 15:46:01.209529 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.209262 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-trusted-ca\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.209529 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.209472 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/61cb38d8-9f5e-495b-931b-7561758ce3a3-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-jp9sz\" (UID: \"61cb38d8-9f5e-495b-931b-7561758ce3a3\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" Jul 24 15:46:01.212305 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.212278 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8d07fcc0-3b9f-45f4-8e85-694a9ef1483f-serving-cert\") pod \"service-ca-operator-5d6c75f84f-wvx4n\" (UID: \"8d07fcc0-3b9f-45f4-8e85-694a9ef1483f\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wvx4n" Jul 24 15:46:01.212305 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.212289 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-image-registry-private-configuration\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.212483 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.212325 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-installation-pull-secrets\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.218808 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.218780 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-bound-sa-token\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.218992 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.218971 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w6nq\" (UniqueName: \"kubernetes.io/projected/c2185c30-e6e6-4623-9bcb-1a0b6baf046d-kube-api-access-2w6nq\") pod \"volume-data-source-validator-549c6d54cf-d8p9n\" (UID: \"c2185c30-e6e6-4623-9bcb-1a0b6baf046d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d8p9n" Jul 24 15:46:01.219046 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.219030 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jfjh\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-kube-api-access-6jfjh\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.219184 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.219171 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8svgm\" (UniqueName: \"kubernetes.io/projected/61cb38d8-9f5e-495b-931b-7561758ce3a3-kube-api-access-8svgm\") pod \"cluster-monitoring-operator-548b489cb8-jp9sz\" (UID: \"61cb38d8-9f5e-495b-931b-7561758ce3a3\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" Jul 24 15:46:01.228098 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.228070 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfnkw\" (UniqueName: \"kubernetes.io/projected/8d07fcc0-3b9f-45f4-8e85-694a9ef1483f-kube-api-access-rfnkw\") pod \"service-ca-operator-5d6c75f84f-wvx4n\" (UID: \"8d07fcc0-3b9f-45f4-8e85-694a9ef1483f\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wvx4n" Jul 24 15:46:01.276067 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.276030 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d8p9n" Jul 24 15:46:01.309113 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309086 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70e5d390-4438-46a0-9e53-fae9badc534f-trusted-ca\") pod \"console-operator-584dcb6b54-285sz\" (UID: \"70e5d390-4438-46a0-9e53-fae9badc534f\") " pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:01.309248 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309125 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr5rx\" (UniqueName: \"kubernetes.io/projected/70e5d390-4438-46a0-9e53-fae9badc534f-kube-api-access-jr5rx\") pod \"console-operator-584dcb6b54-285sz\" (UID: \"70e5d390-4438-46a0-9e53-fae9badc534f\") " pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:01.309248 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309163 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2397af02-6f7f-4fde-905b-f733e7a77c47-service-ca-bundle\") pod \"insights-operator-5885fdb759-zf2c2\" (UID: \"2397af02-6f7f-4fde-905b-f733e7a77c47\") " pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.309248 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309181 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5zzlc\" (UID: \"ba4a3f79-a5dc-4a5d-b891-38bc267aed63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc" Jul 24 15:46:01.309248 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309198 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2397af02-6f7f-4fde-905b-f733e7a77c47-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-zf2c2\" (UID: \"2397af02-6f7f-4fde-905b-f733e7a77c47\") " pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.309428 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.309315 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:46:01.309428 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.309363 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-samples-operator-tls podName:ba4a3f79-a5dc-4a5d-b891-38bc267aed63 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:01.809349474 +0000 UTC m=+36.727319752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-samples-operator-tls") pod "cluster-samples-operator-6748489656-5zzlc" (UID: "ba4a3f79-a5dc-4a5d-b891-38bc267aed63") : secret "samples-operator-tls" not found Jul 24 15:46:01.309428 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309355 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d44kg\" (UID: \"2f97027f-5262-48c0-acd9-61ccc7b95bbf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d44kg" Jul 24 15:46:01.309428 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309393 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2f97027f-5262-48c0-acd9-61ccc7b95bbf-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-d44kg\" (UID: \"2f97027f-5262-48c0-acd9-61ccc7b95bbf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d44kg" Jul 24 15:46:01.309428 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309425 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-stats-auth\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:01.309727 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.309454 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 15:46:01.309727 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.309532 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert podName:2f97027f-5262-48c0-acd9-61ccc7b95bbf nodeName:}" failed. No retries permitted until 2026-07-24 15:46:01.809513574 +0000 UTC m=+36.727483868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-d44kg" (UID: "2f97027f-5262-48c0-acd9-61ccc7b95bbf") : secret "networking-console-plugin-cert" not found Jul 24 15:46:01.309727 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309458 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-default-certificate\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:01.309727 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309578 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70e5d390-4438-46a0-9e53-fae9badc534f-serving-cert\") pod \"console-operator-584dcb6b54-285sz\" (UID: \"70e5d390-4438-46a0-9e53-fae9badc534f\") " pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:01.309727 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309613 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29t6s\" (UniqueName: \"kubernetes.io/projected/c8a9959c-15b2-499e-a9ff-0eb605fa149c-kube-api-access-29t6s\") pod \"network-check-source-7d88cb55d8-xg7vg\" (UID: \"c8a9959c-15b2-499e-a9ff-0eb605fa149c\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xg7vg" Jul 24 15:46:01.309727 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309642 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czrnr\" (UniqueName: \"kubernetes.io/projected/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-kube-api-access-czrnr\") pod \"cluster-samples-operator-6748489656-5zzlc\" (UID: \"ba4a3f79-a5dc-4a5d-b891-38bc267aed63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc" Jul 24 15:46:01.309727 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309670 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2397af02-6f7f-4fde-905b-f733e7a77c47-snapshots\") pod \"insights-operator-5885fdb759-zf2c2\" (UID: \"2397af02-6f7f-4fde-905b-f733e7a77c47\") " pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.309727 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309695 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2397af02-6f7f-4fde-905b-f733e7a77c47-serving-cert\") pod \"insights-operator-5885fdb759-zf2c2\" (UID: \"2397af02-6f7f-4fde-905b-f733e7a77c47\") " pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.309727 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309721 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg6l5\" (UniqueName: \"kubernetes.io/projected/2397af02-6f7f-4fde-905b-f733e7a77c47-kube-api-access-vg6l5\") pod \"insights-operator-5885fdb759-zf2c2\" (UID: \"2397af02-6f7f-4fde-905b-f733e7a77c47\") " pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.310149 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309747 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8h4w\" (UniqueName: \"kubernetes.io/projected/20a41a97-5d30-42f9-8b18-138802f0f554-kube-api-access-w8h4w\") pod \"dns-default-fwkmh\" (UID: \"20a41a97-5d30-42f9-8b18-138802f0f554\") " pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:01.310149 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309781 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a41a97-5d30-42f9-8b18-138802f0f554-tmp-dir\") pod \"dns-default-fwkmh\" (UID: \"20a41a97-5d30-42f9-8b18-138802f0f554\") " pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:01.310149 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309807 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2397af02-6f7f-4fde-905b-f733e7a77c47-service-ca-bundle\") pod \"insights-operator-5885fdb759-zf2c2\" (UID: \"2397af02-6f7f-4fde-905b-f733e7a77c47\") " pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.310149 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309827 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44t8w\" (UniqueName: \"kubernetes.io/projected/fe32eca6-90f7-4d13-a58f-156b373a18f9-kube-api-access-44t8w\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:01.310149 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309885 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea54c847-2195-4b03-9e9d-31efaf46de3f-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m\" (UID: \"ea54c847-2195-4b03-9e9d-31efaf46de3f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m" Jul 24 15:46:01.310149 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309918 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:01.310149 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.309949 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:01.310149 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.310003 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch2nn\" (UniqueName: \"kubernetes.io/projected/ea54c847-2195-4b03-9e9d-31efaf46de3f-kube-api-access-ch2nn\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m\" (UID: \"ea54c847-2195-4b03-9e9d-31efaf46de3f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m" Jul 24 15:46:01.310149 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.310028 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls\") pod \"dns-default-fwkmh\" (UID: \"20a41a97-5d30-42f9-8b18-138802f0f554\") " pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:01.310149 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.310069 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2397af02-6f7f-4fde-905b-f733e7a77c47-tmp\") pod \"insights-operator-5885fdb759-zf2c2\" (UID: \"2397af02-6f7f-4fde-905b-f733e7a77c47\") " pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.310149 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.310097 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6phv\" (UniqueName: \"kubernetes.io/projected/d6d6c0cc-0d5f-407c-b9ea-00cf14c8076e-kube-api-access-x6phv\") pod \"ingress-canary-6v2fh\" (UID: \"d6d6c0cc-0d5f-407c-b9ea-00cf14c8076e\") " pod="openshift-ingress-canary/ingress-canary-6v2fh" Jul 24 15:46:01.310149 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.310122 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/20a41a97-5d30-42f9-8b18-138802f0f554-config-volume\") pod \"dns-default-fwkmh\" (UID: \"20a41a97-5d30-42f9-8b18-138802f0f554\") " pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:01.310719 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.310164 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea54c847-2195-4b03-9e9d-31efaf46de3f-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m\" (UID: \"ea54c847-2195-4b03-9e9d-31efaf46de3f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m" Jul 24 15:46:01.310719 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.310192 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70e5d390-4438-46a0-9e53-fae9badc534f-config\") pod \"console-operator-584dcb6b54-285sz\" (UID: \"70e5d390-4438-46a0-9e53-fae9badc534f\") " pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:01.310719 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.310244 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6d6c0cc-0d5f-407c-b9ea-00cf14c8076e-cert\") pod \"ingress-canary-6v2fh\" (UID: \"d6d6c0cc-0d5f-407c-b9ea-00cf14c8076e\") " pod="openshift-ingress-canary/ingress-canary-6v2fh" Jul 24 15:46:01.310719 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.310253 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2f97027f-5262-48c0-acd9-61ccc7b95bbf-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-d44kg\" (UID: \"2f97027f-5262-48c0-acd9-61ccc7b95bbf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d44kg" Jul 24 15:46:01.310719 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.310369 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70e5d390-4438-46a0-9e53-fae9badc534f-trusted-ca\") pod \"console-operator-584dcb6b54-285sz\" (UID: \"70e5d390-4438-46a0-9e53-fae9badc534f\") " pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:01.310719 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.310621 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2397af02-6f7f-4fde-905b-f733e7a77c47-tmp\") pod \"insights-operator-5885fdb759-zf2c2\" (UID: \"2397af02-6f7f-4fde-905b-f733e7a77c47\") " pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.311002 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.310781 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2397af02-6f7f-4fde-905b-f733e7a77c47-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-zf2c2\" (UID: \"2397af02-6f7f-4fde-905b-f733e7a77c47\") " pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.311002 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.310831 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ea54c847-2195-4b03-9e9d-31efaf46de3f-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m\" (UID: \"ea54c847-2195-4b03-9e9d-31efaf46de3f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m" Jul 24 15:46:01.311148 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.311131 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2397af02-6f7f-4fde-905b-f733e7a77c47-snapshots\") pod \"insights-operator-5885fdb759-zf2c2\" (UID: \"2397af02-6f7f-4fde-905b-f733e7a77c47\") " pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.311330 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.311309 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70e5d390-4438-46a0-9e53-fae9badc534f-config\") pod \"console-operator-584dcb6b54-285sz\" (UID: \"70e5d390-4438-46a0-9e53-fae9badc534f\") " pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:01.312083 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.312065 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wvx4n" Jul 24 15:46:01.312754 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.312735 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2397af02-6f7f-4fde-905b-f733e7a77c47-serving-cert\") pod \"insights-operator-5885fdb759-zf2c2\" (UID: \"2397af02-6f7f-4fde-905b-f733e7a77c47\") " pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.313081 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.313064 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/70e5d390-4438-46a0-9e53-fae9badc534f-serving-cert\") pod \"console-operator-584dcb6b54-285sz\" (UID: \"70e5d390-4438-46a0-9e53-fae9badc534f\") " pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:01.313146 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.313087 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea54c847-2195-4b03-9e9d-31efaf46de3f-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m\" (UID: \"ea54c847-2195-4b03-9e9d-31efaf46de3f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m" Jul 24 15:46:01.318472 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.318450 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr5rx\" (UniqueName: \"kubernetes.io/projected/70e5d390-4438-46a0-9e53-fae9badc534f-kube-api-access-jr5rx\") pod \"console-operator-584dcb6b54-285sz\" (UID: \"70e5d390-4438-46a0-9e53-fae9badc534f\") " pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:01.318950 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.318922 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg6l5\" (UniqueName: \"kubernetes.io/projected/2397af02-6f7f-4fde-905b-f733e7a77c47-kube-api-access-vg6l5\") pod \"insights-operator-5885fdb759-zf2c2\" (UID: \"2397af02-6f7f-4fde-905b-f733e7a77c47\") " pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.319019 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.318932 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czrnr\" (UniqueName: \"kubernetes.io/projected/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-kube-api-access-czrnr\") pod \"cluster-samples-operator-6748489656-5zzlc\" (UID: \"ba4a3f79-a5dc-4a5d-b891-38bc267aed63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc" Jul 24 15:46:01.319195 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.319180 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch2nn\" (UniqueName: \"kubernetes.io/projected/ea54c847-2195-4b03-9e9d-31efaf46de3f-kube-api-access-ch2nn\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m\" (UID: \"ea54c847-2195-4b03-9e9d-31efaf46de3f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m" Jul 24 15:46:01.334198 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.334162 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m" Jul 24 15:46:01.383603 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.383568 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-zf2c2" Jul 24 15:46:01.408629 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.408579 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:01.410877 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.410855 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-stats-auth\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:01.410959 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.410893 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-default-certificate\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:01.410959 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.410918 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29t6s\" (UniqueName: \"kubernetes.io/projected/c8a9959c-15b2-499e-a9ff-0eb605fa149c-kube-api-access-29t6s\") pod \"network-check-source-7d88cb55d8-xg7vg\" (UID: \"c8a9959c-15b2-499e-a9ff-0eb605fa149c\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xg7vg" Jul 24 15:46:01.411082 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.411061 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8h4w\" (UniqueName: \"kubernetes.io/projected/20a41a97-5d30-42f9-8b18-138802f0f554-kube-api-access-w8h4w\") pod \"dns-default-fwkmh\" (UID: \"20a41a97-5d30-42f9-8b18-138802f0f554\") " pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:01.411136 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.411103 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a41a97-5d30-42f9-8b18-138802f0f554-tmp-dir\") pod \"dns-default-fwkmh\" (UID: \"20a41a97-5d30-42f9-8b18-138802f0f554\") " pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:01.411180 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.411139 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44t8w\" (UniqueName: \"kubernetes.io/projected/fe32eca6-90f7-4d13-a58f-156b373a18f9-kube-api-access-44t8w\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:01.411180 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.411168 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:01.411302 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.411196 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:01.411302 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.411258 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls\") pod \"dns-default-fwkmh\" (UID: \"20a41a97-5d30-42f9-8b18-138802f0f554\") " pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:01.411402 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.411308 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6phv\" (UniqueName: \"kubernetes.io/projected/d6d6c0cc-0d5f-407c-b9ea-00cf14c8076e-kube-api-access-x6phv\") pod \"ingress-canary-6v2fh\" (UID: \"d6d6c0cc-0d5f-407c-b9ea-00cf14c8076e\") " pod="openshift-ingress-canary/ingress-canary-6v2fh" Jul 24 15:46:01.411402 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.411337 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/20a41a97-5d30-42f9-8b18-138802f0f554-config-volume\") pod \"dns-default-fwkmh\" (UID: \"20a41a97-5d30-42f9-8b18-138802f0f554\") " pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:01.411402 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.411374 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6d6c0cc-0d5f-407c-b9ea-00cf14c8076e-cert\") pod \"ingress-canary-6v2fh\" (UID: \"d6d6c0cc-0d5f-407c-b9ea-00cf14c8076e\") " pod="openshift-ingress-canary/ingress-canary-6v2fh" Jul 24 15:46:01.411540 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.411431 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle podName:fe32eca6-90f7-4d13-a58f-156b373a18f9 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:01.911409114 +0000 UTC m=+36.829379396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle") pod "router-default-65b698c97-9vtdf" (UID: "fe32eca6-90f7-4d13-a58f-156b373a18f9") : configmap references non-existent config key: service-ca.crt Jul 24 15:46:01.411540 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.411457 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20a41a97-5d30-42f9-8b18-138802f0f554-tmp-dir\") pod \"dns-default-fwkmh\" (UID: \"20a41a97-5d30-42f9-8b18-138802f0f554\") " pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:01.411647 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.411566 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 15:46:01.411647 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.411621 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs podName:fe32eca6-90f7-4d13-a58f-156b373a18f9 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:01.911602543 +0000 UTC m=+36.829572824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs") pod "router-default-65b698c97-9vtdf" (UID: "fe32eca6-90f7-4d13-a58f-156b373a18f9") : secret "router-metrics-certs-default" not found Jul 24 15:46:01.411746 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.411681 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:46:01.411746 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.411716 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls podName:20a41a97-5d30-42f9-8b18-138802f0f554 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:01.911703904 +0000 UTC m=+36.829674196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls") pod "dns-default-fwkmh" (UID: "20a41a97-5d30-42f9-8b18-138802f0f554") : secret "dns-default-metrics-tls" not found Jul 24 15:46:01.412024 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.411994 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/20a41a97-5d30-42f9-8b18-138802f0f554-config-volume\") pod \"dns-default-fwkmh\" (UID: \"20a41a97-5d30-42f9-8b18-138802f0f554\") " pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:01.414040 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.414019 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6d6c0cc-0d5f-407c-b9ea-00cf14c8076e-cert\") pod \"ingress-canary-6v2fh\" (UID: \"d6d6c0cc-0d5f-407c-b9ea-00cf14c8076e\") " pod="openshift-ingress-canary/ingress-canary-6v2fh" Jul 24 15:46:01.414172 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.414151 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-default-certificate\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:01.414328 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.414306 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-stats-auth\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:01.421393 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.421368 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6phv\" (UniqueName: \"kubernetes.io/projected/d6d6c0cc-0d5f-407c-b9ea-00cf14c8076e-kube-api-access-x6phv\") pod \"ingress-canary-6v2fh\" (UID: \"d6d6c0cc-0d5f-407c-b9ea-00cf14c8076e\") " pod="openshift-ingress-canary/ingress-canary-6v2fh" Jul 24 15:46:01.421482 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.421464 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8h4w\" (UniqueName: \"kubernetes.io/projected/20a41a97-5d30-42f9-8b18-138802f0f554-kube-api-access-w8h4w\") pod \"dns-default-fwkmh\" (UID: \"20a41a97-5d30-42f9-8b18-138802f0f554\") " pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:01.421569 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.421550 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29t6s\" (UniqueName: \"kubernetes.io/projected/c8a9959c-15b2-499e-a9ff-0eb605fa149c-kube-api-access-29t6s\") pod \"network-check-source-7d88cb55d8-xg7vg\" (UID: \"c8a9959c-15b2-499e-a9ff-0eb605fa149c\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xg7vg" Jul 24 15:46:01.421824 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.421809 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44t8w\" (UniqueName: \"kubernetes.io/projected/fe32eca6-90f7-4d13-a58f-156b373a18f9-kube-api-access-44t8w\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:01.437156 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.437128 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6v2fh" Jul 24 15:46:01.441517 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.441492 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d8p9n"] Jul 24 15:46:01.445554 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:01.445521 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2185c30_e6e6_4623_9bcb_1a0b6baf046d.slice/crio-41a58e80bf4abc6f1d7926271fd2444ca681fe8b80e7853460cdab24470377e5 WatchSource:0}: Error finding container 41a58e80bf4abc6f1d7926271fd2444ca681fe8b80e7853460cdab24470377e5: Status 404 returned error can't find the container with id 41a58e80bf4abc6f1d7926271fd2444ca681fe8b80e7853460cdab24470377e5 Jul 24 15:46:01.469242 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.469093 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xg7vg" Jul 24 15:46:01.625585 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.624250 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:46:01.632378 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.627913 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 15:46:01.651892 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.651864 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wvx4n"] Jul 24 15:46:01.660398 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:01.660367 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d07fcc0_3b9f_45f4_8e85_694a9ef1483f.slice/crio-f930784b64dded2d0e4372dc418e45177de8e3bc05d277358b2ece800e6c15da WatchSource:0}: Error finding container f930784b64dded2d0e4372dc418e45177de8e3bc05d277358b2ece800e6c15da: Status 404 returned error can't find the container with id f930784b64dded2d0e4372dc418e45177de8e3bc05d277358b2ece800e6c15da Jul 24 15:46:01.667795 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.667718 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m"] Jul 24 15:46:01.671150 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:01.671124 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea54c847_2195_4b03_9e9d_31efaf46de3f.slice/crio-4f3db77a7b2e31748702c5d6c194f6d2dba2baa459f509dec55d0dc71416a861 WatchSource:0}: Error finding container 4f3db77a7b2e31748702c5d6c194f6d2dba2baa459f509dec55d0dc71416a861: Status 404 returned error can't find the container with id 4f3db77a7b2e31748702c5d6c194f6d2dba2baa459f509dec55d0dc71416a861 Jul 24 15:46:01.678534 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.678471 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-zf2c2"] Jul 24 15:46:01.682770 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:01.682739 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2397af02_6f7f_4fde_905b_f733e7a77c47.slice/crio-1f42c1781213d2954c799a019f3dc65f40068ed97d2996468f04f8dbc66f6557 WatchSource:0}: Error finding container 1f42c1781213d2954c799a019f3dc65f40068ed97d2996468f04f8dbc66f6557: Status 404 returned error can't find the container with id 1f42c1781213d2954c799a019f3dc65f40068ed97d2996468f04f8dbc66f6557 Jul 24 15:46:01.688626 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.688597 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-285sz"] Jul 24 15:46:01.693377 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.693355 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6v2fh"] Jul 24 15:46:01.715374 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.715347 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jp9sz\" (UID: \"61cb38d8-9f5e-495b-931b-7561758ce3a3\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" Jul 24 15:46:01.715527 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.715458 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:01.715527 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.715495 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 15:46:01.715594 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.715561 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls podName:61cb38d8-9f5e-495b-931b-7561758ce3a3 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:02.715544315 +0000 UTC m=+37.633514594 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jp9sz" (UID: "61cb38d8-9f5e-495b-931b-7561758ce3a3") : secret "cluster-monitoring-operator-tls" not found Jul 24 15:46:01.715594 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.715565 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:46:01.715594 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.715575 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d75669b64-9vf6v: secret "image-registry-tls" not found Jul 24 15:46:01.715684 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.715609 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls podName:6555d7aa-ea7e-4dbf-8bb1-be3e103afc39 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:02.715598393 +0000 UTC m=+37.633568672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls") pod "image-registry-6d75669b64-9vf6v" (UID: "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39") : secret "image-registry-tls" not found Jul 24 15:46:01.723295 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.723271 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-xg7vg"] Jul 24 15:46:01.726644 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:01.726621 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a9959c_15b2_499e_a9ff_0eb605fa149c.slice/crio-cbb2db084ad5cca91bcb295b23f2583b85f3033b560e64b8aa17a1f3bad776a5 WatchSource:0}: Error finding container cbb2db084ad5cca91bcb295b23f2583b85f3033b560e64b8aa17a1f3bad776a5: Status 404 returned error can't find the container with id cbb2db084ad5cca91bcb295b23f2583b85f3033b560e64b8aa17a1f3bad776a5 Jul 24 15:46:01.727955 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:01.727913 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e5d390_4438_46a0_9e53_fae9badc534f.slice/crio-fa6a8c41bc92a4da06d8ed813bbe08e702fbbc59a4c7ad13118f671ed3881dcc WatchSource:0}: Error finding container fa6a8c41bc92a4da06d8ed813bbe08e702fbbc59a4c7ad13118f671ed3881dcc: Status 404 returned error can't find the container with id fa6a8c41bc92a4da06d8ed813bbe08e702fbbc59a4c7ad13118f671ed3881dcc Jul 24 15:46:01.729433 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:01.729340 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d6c0cc_0d5f_407c_b9ea_00cf14c8076e.slice/crio-348f3cb9c4eb70dbc986bdecdff6601081ee480165a5d503bf12cc192aa83178 WatchSource:0}: Error finding container 348f3cb9c4eb70dbc986bdecdff6601081ee480165a5d503bf12cc192aa83178: Status 404 returned error can't find the container with id 348f3cb9c4eb70dbc986bdecdff6601081ee480165a5d503bf12cc192aa83178 Jul 24 15:46:01.810336 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.810301 2569 generic.go:356] "Generic (PLEG): container finished" podID="6f58d4fe-4a3c-466c-8001-85fd8797a32a" containerID="397aa4c40a8f6e0c8b9d9c8ef9b4ac8ec690cd0d5235ac662a14ee509d8af574" exitCode=0 Jul 24 15:46:01.810504 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.810386 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkwbb" event={"ID":"6f58d4fe-4a3c-466c-8001-85fd8797a32a","Type":"ContainerDied","Data":"397aa4c40a8f6e0c8b9d9c8ef9b4ac8ec690cd0d5235ac662a14ee509d8af574"} Jul 24 15:46:01.811581 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.811555 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" event={"ID":"70e5d390-4438-46a0-9e53-fae9badc534f","Type":"ContainerStarted","Data":"fa6a8c41bc92a4da06d8ed813bbe08e702fbbc59a4c7ad13118f671ed3881dcc"} Jul 24 15:46:01.812623 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.812601 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d8p9n" event={"ID":"c2185c30-e6e6-4623-9bcb-1a0b6baf046d","Type":"ContainerStarted","Data":"41a58e80bf4abc6f1d7926271fd2444ca681fe8b80e7853460cdab24470377e5"} Jul 24 15:46:01.813634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.813613 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xg7vg" event={"ID":"c8a9959c-15b2-499e-a9ff-0eb605fa149c","Type":"ContainerStarted","Data":"cbb2db084ad5cca91bcb295b23f2583b85f3033b560e64b8aa17a1f3bad776a5"} Jul 24 15:46:01.814766 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.814744 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6v2fh" event={"ID":"d6d6c0cc-0d5f-407c-b9ea-00cf14c8076e","Type":"ContainerStarted","Data":"348f3cb9c4eb70dbc986bdecdff6601081ee480165a5d503bf12cc192aa83178"} Jul 24 15:46:01.818959 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.816498 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5zzlc\" (UID: \"ba4a3f79-a5dc-4a5d-b891-38bc267aed63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc" Jul 24 15:46:01.818959 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.816558 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d44kg\" (UID: \"2f97027f-5262-48c0-acd9-61ccc7b95bbf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d44kg" Jul 24 15:46:01.818959 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.817443 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:46:01.818959 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.817509 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-samples-operator-tls podName:ba4a3f79-a5dc-4a5d-b891-38bc267aed63 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:02.817490482 +0000 UTC m=+37.735460762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-samples-operator-tls") pod "cluster-samples-operator-6748489656-5zzlc" (UID: "ba4a3f79-a5dc-4a5d-b891-38bc267aed63") : secret "samples-operator-tls" not found Jul 24 15:46:01.818959 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.817651 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 15:46:01.818959 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.817708 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert podName:2f97027f-5262-48c0-acd9-61ccc7b95bbf nodeName:}" failed. No retries permitted until 2026-07-24 15:46:02.81769171 +0000 UTC m=+37.735661991 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-d44kg" (UID: "2f97027f-5262-48c0-acd9-61ccc7b95bbf") : secret "networking-console-plugin-cert" not found Jul 24 15:46:01.820074 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.820052 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-zf2c2" event={"ID":"2397af02-6f7f-4fde-905b-f733e7a77c47","Type":"ContainerStarted","Data":"1f42c1781213d2954c799a019f3dc65f40068ed97d2996468f04f8dbc66f6557"} Jul 24 15:46:01.821033 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.821015 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m" event={"ID":"ea54c847-2195-4b03-9e9d-31efaf46de3f","Type":"ContainerStarted","Data":"4f3db77a7b2e31748702c5d6c194f6d2dba2baa459f509dec55d0dc71416a861"} Jul 24 15:46:01.822305 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.822284 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wvx4n" event={"ID":"8d07fcc0-3b9f-45f4-8e85-694a9ef1483f","Type":"ContainerStarted","Data":"f930784b64dded2d0e4372dc418e45177de8e3bc05d277358b2ece800e6c15da"} Jul 24 15:46:01.918094 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.918029 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:01.918094 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.918066 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:01.918244 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:01.918095 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls\") pod \"dns-default-fwkmh\" (UID: \"20a41a97-5d30-42f9-8b18-138802f0f554\") " pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:01.918244 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.918191 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle podName:fe32eca6-90f7-4d13-a58f-156b373a18f9 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:02.918173834 +0000 UTC m=+37.836144116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle") pod "router-default-65b698c97-9vtdf" (UID: "fe32eca6-90f7-4d13-a58f-156b373a18f9") : configmap references non-existent config key: service-ca.crt Jul 24 15:46:01.918331 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.918274 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:46:01.918331 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.918329 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls podName:20a41a97-5d30-42f9-8b18-138802f0f554 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:02.918312609 +0000 UTC m=+37.836282902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls") pod "dns-default-fwkmh" (UID: "20a41a97-5d30-42f9-8b18-138802f0f554") : secret "dns-default-metrics-tls" not found Jul 24 15:46:01.918715 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.918205 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 15:46:01.918776 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:01.918765 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs podName:fe32eca6-90f7-4d13-a58f-156b373a18f9 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:02.918747335 +0000 UTC m=+37.836717633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs") pod "router-default-65b698c97-9vtdf" (UID: "fe32eca6-90f7-4d13-a58f-156b373a18f9") : secret "router-metrics-certs-default" not found Jul 24 15:46:02.622377 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:02.621691 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:46:02.622377 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:02.622172 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:46:02.627835 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:02.627624 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 15:46:02.631257 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:02.630111 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pqxfk\"" Jul 24 15:46:02.631257 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:02.630383 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cbp7q\"" Jul 24 15:46:02.730196 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:02.728940 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jp9sz\" (UID: \"61cb38d8-9f5e-495b-931b-7561758ce3a3\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" Jul 24 15:46:02.730196 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:02.729109 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:02.730196 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:02.729283 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:46:02.730196 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:02.729299 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d75669b64-9vf6v: secret "image-registry-tls" not found Jul 24 15:46:02.730196 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:02.729357 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls podName:6555d7aa-ea7e-4dbf-8bb1-be3e103afc39 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:04.729336985 +0000 UTC m=+39.647307269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls") pod "image-registry-6d75669b64-9vf6v" (UID: "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39") : secret "image-registry-tls" not found Jul 24 15:46:02.730196 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:02.729425 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 15:46:02.730196 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:02.729460 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls podName:61cb38d8-9f5e-495b-931b-7561758ce3a3 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:04.729448984 +0000 UTC m=+39.647419270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jp9sz" (UID: "61cb38d8-9f5e-495b-931b-7561758ce3a3") : secret "cluster-monitoring-operator-tls" not found Jul 24 15:46:02.829758 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:02.829722 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5zzlc\" (UID: \"ba4a3f79-a5dc-4a5d-b891-38bc267aed63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc" Jul 24 15:46:02.829953 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:02.829773 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d44kg\" (UID: \"2f97027f-5262-48c0-acd9-61ccc7b95bbf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d44kg" Jul 24 15:46:02.829953 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:02.829935 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 15:46:02.830068 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:02.829996 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert podName:2f97027f-5262-48c0-acd9-61ccc7b95bbf nodeName:}" failed. No retries permitted until 2026-07-24 15:46:04.829978008 +0000 UTC m=+39.747948290 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-d44kg" (UID: "2f97027f-5262-48c0-acd9-61ccc7b95bbf") : secret "networking-console-plugin-cert" not found Jul 24 15:46:02.830427 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:02.830406 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:46:02.830518 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:02.830467 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-samples-operator-tls podName:ba4a3f79-a5dc-4a5d-b891-38bc267aed63 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:04.830449969 +0000 UTC m=+39.748420252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-samples-operator-tls") pod "cluster-samples-operator-6748489656-5zzlc" (UID: "ba4a3f79-a5dc-4a5d-b891-38bc267aed63") : secret "samples-operator-tls" not found Jul 24 15:46:02.840148 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:02.840114 2569 generic.go:356] "Generic (PLEG): container finished" podID="6f58d4fe-4a3c-466c-8001-85fd8797a32a" containerID="6fc5dd3b53e0d35bab2af6adc3c0ebe85ddef95d6ae2d33cd0a5566dde36e354" exitCode=0 Jul 24 15:46:02.840317 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:02.840201 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkwbb" event={"ID":"6f58d4fe-4a3c-466c-8001-85fd8797a32a","Type":"ContainerDied","Data":"6fc5dd3b53e0d35bab2af6adc3c0ebe85ddef95d6ae2d33cd0a5566dde36e354"} Jul 24 15:46:02.931501 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:02.931288 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:02.931501 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:02.931342 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:02.931501 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:02.931404 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls\") pod \"dns-default-fwkmh\" (UID: \"20a41a97-5d30-42f9-8b18-138802f0f554\") " pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:02.931798 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:02.931591 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:46:02.931798 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:02.931658 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls podName:20a41a97-5d30-42f9-8b18-138802f0f554 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:04.931638913 +0000 UTC m=+39.849609210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls") pod "dns-default-fwkmh" (UID: "20a41a97-5d30-42f9-8b18-138802f0f554") : secret "dns-default-metrics-tls" not found Jul 24 15:46:02.932752 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:02.932536 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle podName:fe32eca6-90f7-4d13-a58f-156b373a18f9 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:04.932517323 +0000 UTC m=+39.850487715 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle") pod "router-default-65b698c97-9vtdf" (UID: "fe32eca6-90f7-4d13-a58f-156b373a18f9") : configmap references non-existent config key: service-ca.crt Jul 24 15:46:02.933479 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:02.933460 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 15:46:02.933584 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:02.933557 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs podName:fe32eca6-90f7-4d13-a58f-156b373a18f9 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:04.933541757 +0000 UTC m=+39.851512038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs") pod "router-default-65b698c97-9vtdf" (UID: "fe32eca6-90f7-4d13-a58f-156b373a18f9") : secret "router-metrics-certs-default" not found Jul 24 15:46:03.859956 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:03.859071 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lkwbb" event={"ID":"6f58d4fe-4a3c-466c-8001-85fd8797a32a","Type":"ContainerStarted","Data":"8e6e149fbc165d539f6f665859f908b3649e95d5f021f0a8636d244aff9da32a"} Jul 24 15:46:03.884712 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:03.883966 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lkwbb" podStartSLOduration=9.087514967 podStartE2EDuration="38.883944863s" totalImagesPullingTime="29.796429896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:45:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:03.883764762 +0000 UTC m=+38.801735054" watchObservedRunningTime="2026-07-24 15:46:03.883944863 +0000 UTC m=+38.801915164" Jul 24 15:46:04.750073 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:04.750029 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jp9sz\" (UID: \"61cb38d8-9f5e-495b-931b-7561758ce3a3\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" Jul 24 15:46:04.750284 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:04.750185 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:04.750284 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:04.750233 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 15:46:04.750413 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:04.750296 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:46:04.750413 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:04.750309 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls podName:61cb38d8-9f5e-495b-931b-7561758ce3a3 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:08.750287564 +0000 UTC m=+43.668257858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jp9sz" (UID: "61cb38d8-9f5e-495b-931b-7561758ce3a3") : secret "cluster-monitoring-operator-tls" not found Jul 24 15:46:04.750413 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:04.750310 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d75669b64-9vf6v: secret "image-registry-tls" not found Jul 24 15:46:04.750413 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:04.750352 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls podName:6555d7aa-ea7e-4dbf-8bb1-be3e103afc39 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:08.750342368 +0000 UTC m=+43.668312646 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls") pod "image-registry-6d75669b64-9vf6v" (UID: "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39") : secret "image-registry-tls" not found Jul 24 15:46:04.851608 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:04.851565 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5zzlc\" (UID: \"ba4a3f79-a5dc-4a5d-b891-38bc267aed63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc" Jul 24 15:46:04.851790 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:04.851615 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d44kg\" (UID: \"2f97027f-5262-48c0-acd9-61ccc7b95bbf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d44kg" Jul 24 15:46:04.851790 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:04.851757 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:46:04.851886 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:04.851829 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-samples-operator-tls podName:ba4a3f79-a5dc-4a5d-b891-38bc267aed63 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:08.851811537 +0000 UTC m=+43.769781816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-samples-operator-tls") pod "cluster-samples-operator-6748489656-5zzlc" (UID: "ba4a3f79-a5dc-4a5d-b891-38bc267aed63") : secret "samples-operator-tls" not found Jul 24 15:46:04.852333 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:04.852313 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 15:46:04.852407 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:04.852383 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert podName:2f97027f-5262-48c0-acd9-61ccc7b95bbf nodeName:}" failed. No retries permitted until 2026-07-24 15:46:08.852365613 +0000 UTC m=+43.770335895 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-d44kg" (UID: "2f97027f-5262-48c0-acd9-61ccc7b95bbf") : secret "networking-console-plugin-cert" not found Jul 24 15:46:04.869598 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:04.869555 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d8p9n" event={"ID":"c2185c30-e6e6-4623-9bcb-1a0b6baf046d","Type":"ContainerStarted","Data":"e2f22f6e4faa44dfc22b63ab22908b757bf9472afc36451f6706d9b7d614ec33"} Jul 24 15:46:04.887053 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:04.886999 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d8p9n" podStartSLOduration=18.608757476 podStartE2EDuration="20.886983088s" totalImagesPullingTime="2.278225612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:04.885713583 +0000 UTC m=+39.803683894" watchObservedRunningTime="2026-07-24 15:46:04.886983088 +0000 UTC m=+39.804953388" Jul 24 15:46:04.953070 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:04.953030 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:04.953245 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:04.953170 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle podName:fe32eca6-90f7-4d13-a58f-156b373a18f9 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:08.95313465 +0000 UTC m=+43.871104928 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle") pod "router-default-65b698c97-9vtdf" (UID: "fe32eca6-90f7-4d13-a58f-156b373a18f9") : configmap references non-existent config key: service-ca.crt Jul 24 15:46:04.953245 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:04.953193 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:04.953354 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:04.953272 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls\") pod \"dns-default-fwkmh\" (UID: \"20a41a97-5d30-42f9-8b18-138802f0f554\") " pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:04.953574 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:04.953550 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:46:04.953702 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:04.953594 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls podName:20a41a97-5d30-42f9-8b18-138802f0f554 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:08.953580254 +0000 UTC m=+43.871550533 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls") pod "dns-default-fwkmh" (UID: "20a41a97-5d30-42f9-8b18-138802f0f554") : secret "dns-default-metrics-tls" not found Jul 24 15:46:04.953702 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:04.953621 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 15:46:04.953702 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:04.953678 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs podName:fe32eca6-90f7-4d13-a58f-156b373a18f9 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:08.953662095 +0000 UTC m=+43.871632372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs") pod "router-default-65b698c97-9vtdf" (UID: "fe32eca6-90f7-4d13-a58f-156b373a18f9") : secret "router-metrics-certs-default" not found Jul 24 15:46:08.791191 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:08.791146 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jp9sz\" (UID: \"61cb38d8-9f5e-495b-931b-7561758ce3a3\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" Jul 24 15:46:08.791588 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:08.791297 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:08.791588 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:08.791313 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 15:46:08.791588 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:08.791380 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls podName:61cb38d8-9f5e-495b-931b-7561758ce3a3 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:16.791364558 +0000 UTC m=+51.709334836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jp9sz" (UID: "61cb38d8-9f5e-495b-931b-7561758ce3a3") : secret "cluster-monitoring-operator-tls" not found Jul 24 15:46:08.791588 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:08.791415 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:46:08.791588 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:08.791429 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d75669b64-9vf6v: secret "image-registry-tls" not found Jul 24 15:46:08.791588 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:08.791472 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls podName:6555d7aa-ea7e-4dbf-8bb1-be3e103afc39 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:16.791460717 +0000 UTC m=+51.709430996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls") pod "image-registry-6d75669b64-9vf6v" (UID: "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39") : secret "image-registry-tls" not found Jul 24 15:46:08.892471 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:08.892440 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5zzlc\" (UID: \"ba4a3f79-a5dc-4a5d-b891-38bc267aed63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc" Jul 24 15:46:08.892629 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:08.892478 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d44kg\" (UID: \"2f97027f-5262-48c0-acd9-61ccc7b95bbf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d44kg" Jul 24 15:46:08.892629 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:08.892596 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:46:08.892710 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:08.892649 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 15:46:08.892710 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:08.892654 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-samples-operator-tls podName:ba4a3f79-a5dc-4a5d-b891-38bc267aed63 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:16.892638385 +0000 UTC m=+51.810608664 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-samples-operator-tls") pod "cluster-samples-operator-6748489656-5zzlc" (UID: "ba4a3f79-a5dc-4a5d-b891-38bc267aed63") : secret "samples-operator-tls" not found Jul 24 15:46:08.892710 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:08.892702 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert podName:2f97027f-5262-48c0-acd9-61ccc7b95bbf nodeName:}" failed. No retries permitted until 2026-07-24 15:46:16.892690182 +0000 UTC m=+51.810660459 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-d44kg" (UID: "2f97027f-5262-48c0-acd9-61ccc7b95bbf") : secret "networking-console-plugin-cert" not found Jul 24 15:46:08.993934 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:08.993900 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:08.994036 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:08.993943 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:08.994036 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:08.993979 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls\") pod \"dns-default-fwkmh\" (UID: \"20a41a97-5d30-42f9-8b18-138802f0f554\") " pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:08.994153 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:08.994067 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle podName:fe32eca6-90f7-4d13-a58f-156b373a18f9 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:16.994049393 +0000 UTC m=+51.912019675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle") pod "router-default-65b698c97-9vtdf" (UID: "fe32eca6-90f7-4d13-a58f-156b373a18f9") : configmap references non-existent config key: service-ca.crt Jul 24 15:46:08.994153 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:08.994098 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:46:08.994153 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:08.994117 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 15:46:08.994153 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:08.994139 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls podName:20a41a97-5d30-42f9-8b18-138802f0f554 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:16.994126237 +0000 UTC m=+51.912096515 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls") pod "dns-default-fwkmh" (UID: "20a41a97-5d30-42f9-8b18-138802f0f554") : secret "dns-default-metrics-tls" not found Jul 24 15:46:08.994379 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:08.994173 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs podName:fe32eca6-90f7-4d13-a58f-156b373a18f9 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:16.994156828 +0000 UTC m=+51.912127106 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs") pod "router-default-65b698c97-9vtdf" (UID: "fe32eca6-90f7-4d13-a58f-156b373a18f9") : secret "router-metrics-certs-default" not found Jul 24 15:46:09.885457 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:09.885361 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xg7vg" event={"ID":"c8a9959c-15b2-499e-a9ff-0eb605fa149c","Type":"ContainerStarted","Data":"7796079c7eab53d9baf838f107fd497377aee1223c621a16c02ead2406262822"} Jul 24 15:46:09.887624 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:09.887573 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6v2fh" event={"ID":"d6d6c0cc-0d5f-407c-b9ea-00cf14c8076e","Type":"ContainerStarted","Data":"0e8fc169b41ab04c575016ea0d7bc693a6eef7ce7159e71ae0d2eabcf082faa1"} Jul 24 15:46:09.889191 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:09.889160 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-zf2c2" event={"ID":"2397af02-6f7f-4fde-905b-f733e7a77c47","Type":"ContainerStarted","Data":"4108fd5dbc6a39fef482b86f69fdde7c462372dda3c7ed7eb2787d96787ba56b"} Jul 24 15:46:09.890956 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:09.890934 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m" event={"ID":"ea54c847-2195-4b03-9e9d-31efaf46de3f","Type":"ContainerStarted","Data":"30e0a59abde673b50728e23ac1aa111dabb2979fbae4929b9e4f27b66c9711e5"} Jul 24 15:46:09.892576 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:09.892557 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wvx4n" event={"ID":"8d07fcc0-3b9f-45f4-8e85-694a9ef1483f","Type":"ContainerStarted","Data":"d78575e0b45d4f1a6a95fd4988555cfb512ca38016c8370df69e781900f01229"} Jul 24 15:46:09.894702 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:09.894682 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/0.log" Jul 24 15:46:09.894800 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:09.894722 2569 generic.go:356] "Generic (PLEG): container finished" podID="70e5d390-4438-46a0-9e53-fae9badc534f" containerID="73b93877e897ef0e8c81bf60f63a1a15bfdcffb71961a5717b84983b2ce3dba0" exitCode=255 Jul 24 15:46:09.894800 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:09.894753 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" event={"ID":"70e5d390-4438-46a0-9e53-fae9badc534f","Type":"ContainerDied","Data":"73b93877e897ef0e8c81bf60f63a1a15bfdcffb71961a5717b84983b2ce3dba0"} Jul 24 15:46:09.894985 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:09.894967 2569 scope.go:117] "RemoveContainer" containerID="73b93877e897ef0e8c81bf60f63a1a15bfdcffb71961a5717b84983b2ce3dba0" Jul 24 15:46:09.903003 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:09.902937 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-xg7vg" podStartSLOduration=18.365857185 podStartE2EDuration="25.902922635s" totalImagesPullingTime="7.53706545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:09.902006788 +0000 UTC m=+44.819977090" watchObservedRunningTime="2026-07-24 15:46:09.902922635 +0000 UTC m=+44.820892913" Jul 24 15:46:09.922142 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:09.922091 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-zf2c2" podStartSLOduration=18.351651134 podStartE2EDuration="25.922072251s" totalImagesPullingTime="7.570421117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:09.921015805 +0000 UTC m=+44.838986104" watchObservedRunningTime="2026-07-24 15:46:09.922072251 +0000 UTC m=+44.840042555" Jul 24 15:46:09.959697 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:09.959639 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wvx4n" podStartSLOduration=18.366817741 podStartE2EDuration="25.959617165s" totalImagesPullingTime="7.592799424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:09.937232903 +0000 UTC m=+44.855203204" watchObservedRunningTime="2026-07-24 15:46:09.959617165 +0000 UTC m=+44.877587466" Jul 24 15:46:09.959974 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:09.959947 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m" podStartSLOduration=18.378721963 podStartE2EDuration="25.959938656s" totalImagesPullingTime="7.581216693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:09.958231457 +0000 UTC m=+44.876201749" watchObservedRunningTime="2026-07-24 15:46:09.959938656 +0000 UTC m=+44.877908959" Jul 24 15:46:09.999734 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:09.999347 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6v2fh" podStartSLOduration=2.475412934 podStartE2EDuration="9.999325221s" totalImagesPullingTime="7.523912287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:09.997725298 +0000 UTC m=+44.915695596" watchObservedRunningTime="2026-07-24 15:46:09.999325221 +0000 UTC m=+44.917295529" Jul 24 15:46:10.899681 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:10.899655 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/1.log" Jul 24 15:46:10.900169 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:10.900104 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/0.log" Jul 24 15:46:10.900169 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:10.900150 2569 generic.go:356] "Generic (PLEG): container finished" podID="70e5d390-4438-46a0-9e53-fae9badc534f" containerID="93ceb84cce2a1961901a3a8da5ff51ec6c0119b2c6f168605541f3d2183ee0d8" exitCode=255 Jul 24 15:46:10.900315 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:10.900242 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" event={"ID":"70e5d390-4438-46a0-9e53-fae9badc534f","Type":"ContainerDied","Data":"93ceb84cce2a1961901a3a8da5ff51ec6c0119b2c6f168605541f3d2183ee0d8"} Jul 24 15:46:10.900360 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:10.900314 2569 scope.go:117] "RemoveContainer" containerID="73b93877e897ef0e8c81bf60f63a1a15bfdcffb71961a5717b84983b2ce3dba0" Jul 24 15:46:10.900527 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:10.900502 2569 scope.go:117] "RemoveContainer" containerID="93ceb84cce2a1961901a3a8da5ff51ec6c0119b2c6f168605541f3d2183ee0d8" Jul 24 15:46:10.900712 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:10.900690 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-285sz_openshift-console-operator(70e5d390-4438-46a0-9e53-fae9badc534f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" podUID="70e5d390-4438-46a0-9e53-fae9badc534f" Jul 24 15:46:11.408939 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.408901 2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:11.409088 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.408983 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:11.593353 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.593321 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hxpsj"] Jul 24 15:46:11.637402 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.637361 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hxpsj"] Jul 24 15:46:11.637582 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.637499 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:11.640474 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.640448 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 15:46:11.640599 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.640499 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vfsm2\"" Jul 24 15:46:11.640599 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.640579 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 15:46:11.720693 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.720609 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:11.720845 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.720759 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-crio-socket\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:11.720845 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.720797 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vg4v\" (UniqueName: \"kubernetes.io/projected/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-api-access-2vg4v\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:11.720923 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.720857 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-data-volume\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:11.720923 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.720885 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:11.822082 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.822045 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:11.822279 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.822093 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:11.822279 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:11.822160 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:46:11.822279 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:11.822198 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:46:11.822279 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:11.822270 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-insights-runtime-extractor-tls podName:0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:12.322252061 +0000 UTC m=+47.240222339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxpsj" (UID: "0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396") : secret "insights-runtime-extractor-tls" not found Jul 24 15:46:11.822421 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.822338 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-crio-socket\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:11.822421 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.822374 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vg4v\" (UniqueName: \"kubernetes.io/projected/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-api-access-2vg4v\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:11.822421 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:11.822385 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-rbac-proxy-cm podName:0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:12.322368492 +0000 UTC m=+47.240338785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxpsj" (UID: "0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396") : configmap "kube-rbac-proxy" not found Jul 24 15:46:11.822550 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.822489 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-data-volume\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:11.822550 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.822519 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-crio-socket\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:11.822862 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.822845 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-data-volume\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:11.832035 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.832010 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vg4v\" (UniqueName: \"kubernetes.io/projected/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-api-access-2vg4v\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:11.906223 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.906193 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/1.log" Jul 24 15:46:11.906595 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:11.906537 2569 scope.go:117] "RemoveContainer" containerID="93ceb84cce2a1961901a3a8da5ff51ec6c0119b2c6f168605541f3d2183ee0d8" Jul 24 15:46:11.906699 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:11.906680 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-285sz_openshift-console-operator(70e5d390-4438-46a0-9e53-fae9badc534f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" podUID="70e5d390-4438-46a0-9e53-fae9badc534f" Jul 24 15:46:12.326242 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:12.326186 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:12.326449 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:12.326281 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:12.326449 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:12.326358 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:46:12.326449 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:12.326419 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:46:12.326555 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:12.326422 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-insights-runtime-extractor-tls podName:0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:13.326404717 +0000 UTC m=+48.244374998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxpsj" (UID: "0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396") : secret "insights-runtime-extractor-tls" not found Jul 24 15:46:12.326608 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:12.326587 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-rbac-proxy-cm podName:0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:13.32657218 +0000 UTC m=+48.244542458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxpsj" (UID: "0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396") : configmap "kube-rbac-proxy" not found Jul 24 15:46:12.910522 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:12.910486 2569 scope.go:117] "RemoveContainer" containerID="93ceb84cce2a1961901a3a8da5ff51ec6c0119b2c6f168605541f3d2183ee0d8" Jul 24 15:46:12.910981 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:12.910687 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-285sz_openshift-console-operator(70e5d390-4438-46a0-9e53-fae9badc534f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" podUID="70e5d390-4438-46a0-9e53-fae9badc534f" Jul 24 15:46:13.334820 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:13.334727 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:13.334820 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:13.334773 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:13.335061 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:13.334847 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:46:13.335061 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:13.334895 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:46:13.335061 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:13.334904 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-rbac-proxy-cm podName:0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:15.334890521 +0000 UTC m=+50.252860799 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxpsj" (UID: "0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396") : configmap "kube-rbac-proxy" not found Jul 24 15:46:13.335061 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:13.334970 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-insights-runtime-extractor-tls podName:0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:15.334954693 +0000 UTC m=+50.252924970 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxpsj" (UID: "0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396") : secret "insights-runtime-extractor-tls" not found Jul 24 15:46:13.566764 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:13.566738 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bdjkj_3447e518-ab20-411f-aa9f-4918b9246385/dns-node-resolver/0.log" Jul 24 15:46:14.766139 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:14.766109 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hvtww_34e569a7-61a3-4909-865f-36a7bd286efa/node-ca/0.log" Jul 24 15:46:15.352471 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:15.352429 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:15.352647 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:15.352481 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:15.352647 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:15.352549 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:46:15.352647 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:15.352580 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:46:15.352647 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:15.352613 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-rbac-proxy-cm podName:0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:19.352597878 +0000 UTC m=+54.270568156 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxpsj" (UID: "0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396") : configmap "kube-rbac-proxy" not found Jul 24 15:46:15.352777 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:15.352654 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-insights-runtime-extractor-tls podName:0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:19.352635583 +0000 UTC m=+54.270605861 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxpsj" (UID: "0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396") : secret "insights-runtime-extractor-tls" not found Jul 24 15:46:15.366399 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:15.366354 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6v2fh_d6d6c0cc-0d5f-407c-b9ea-00cf14c8076e/serve-healthcheck-canary/0.log" Jul 24 15:46:16.865464 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:16.865429 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:16.865972 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:16.865553 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jp9sz\" (UID: \"61cb38d8-9f5e-495b-931b-7561758ce3a3\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" Jul 24 15:46:16.865972 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:16.865608 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:46:16.865972 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:16.865633 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6d75669b64-9vf6v: secret "image-registry-tls" not found Jul 24 15:46:16.865972 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:16.865693 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 15:46:16.865972 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:16.865701 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls podName:6555d7aa-ea7e-4dbf-8bb1-be3e103afc39 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:32.865680225 +0000 UTC m=+67.783650509 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls") pod "image-registry-6d75669b64-9vf6v" (UID: "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39") : secret "image-registry-tls" not found Jul 24 15:46:16.865972 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:16.865756 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls podName:61cb38d8-9f5e-495b-931b-7561758ce3a3 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:32.86573979 +0000 UTC m=+67.783710070 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jp9sz" (UID: "61cb38d8-9f5e-495b-931b-7561758ce3a3") : secret "cluster-monitoring-operator-tls" not found Jul 24 15:46:16.966127 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:16.966087 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5zzlc\" (UID: \"ba4a3f79-a5dc-4a5d-b891-38bc267aed63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc" Jul 24 15:46:16.966127 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:16.966129 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d44kg\" (UID: \"2f97027f-5262-48c0-acd9-61ccc7b95bbf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d44kg" Jul 24 15:46:16.966403 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:16.966322 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 15:46:16.966459 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:16.966402 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert podName:2f97027f-5262-48c0-acd9-61ccc7b95bbf nodeName:}" failed. No retries permitted until 2026-07-24 15:46:32.966379483 +0000 UTC m=+67.884349763 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-d44kg" (UID: "2f97027f-5262-48c0-acd9-61ccc7b95bbf") : secret "networking-console-plugin-cert" not found Jul 24 15:46:16.980304 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:16.980279 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba4a3f79-a5dc-4a5d-b891-38bc267aed63-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5zzlc\" (UID: \"ba4a3f79-a5dc-4a5d-b891-38bc267aed63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc" Jul 24 15:46:17.067579 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:17.067540 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:17.067579 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:17.067582 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:17.067777 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:17.067625 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls\") pod \"dns-default-fwkmh\" (UID: \"20a41a97-5d30-42f9-8b18-138802f0f554\") " pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:17.067777 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:17.067721 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle podName:fe32eca6-90f7-4d13-a58f-156b373a18f9 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:33.067700678 +0000 UTC m=+67.985670967 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle") pod "router-default-65b698c97-9vtdf" (UID: "fe32eca6-90f7-4d13-a58f-156b373a18f9") : configmap references non-existent config key: service-ca.crt Jul 24 15:46:17.067777 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:17.067737 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:46:17.067777 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:17.067749 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 15:46:17.067903 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:17.067788 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls podName:20a41a97-5d30-42f9-8b18-138802f0f554 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:33.067778065 +0000 UTC m=+67.985748343 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls") pod "dns-default-fwkmh" (UID: "20a41a97-5d30-42f9-8b18-138802f0f554") : secret "dns-default-metrics-tls" not found Jul 24 15:46:17.067903 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:17.067804 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs podName:fe32eca6-90f7-4d13-a58f-156b373a18f9 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:33.06779651 +0000 UTC m=+67.985766791 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs") pod "router-default-65b698c97-9vtdf" (UID: "fe32eca6-90f7-4d13-a58f-156b373a18f9") : secret "router-metrics-certs-default" not found Jul 24 15:46:17.246780 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:17.246749 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc" Jul 24 15:46:17.394509 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:17.394485 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc"] Jul 24 15:46:17.931884 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:17.931851 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc" event={"ID":"ba4a3f79-a5dc-4a5d-b891-38bc267aed63","Type":"ContainerStarted","Data":"fec61e8fa059abf690ce5d4982aaa2940b2b5e7dd154cbc261c40b662d81f088"} Jul 24 15:46:19.391146 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:19.390975 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:19.391146 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:19.391072 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:46:19.391146 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:19.391150 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-rbac-proxy-cm podName:0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:27.391131564 +0000 UTC m=+62.309101861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxpsj" (UID: "0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396") : configmap "kube-rbac-proxy" not found Jul 24 15:46:19.392029 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:19.391266 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:19.392029 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:19.391405 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:46:19.392029 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:19.391445 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-insights-runtime-extractor-tls podName:0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:27.391432358 +0000 UTC m=+62.309402649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxpsj" (UID: "0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396") : secret "insights-runtime-extractor-tls" not found Jul 24 15:46:19.941090 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:19.941060 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc" event={"ID":"ba4a3f79-a5dc-4a5d-b891-38bc267aed63","Type":"ContainerStarted","Data":"1864b23179a733d009f997eba36af0ce57fdfe020fb4d997bc9f95e4f8d9761e"} Jul 24 15:46:19.941289 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:19.941114 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc" event={"ID":"ba4a3f79-a5dc-4a5d-b891-38bc267aed63","Type":"ContainerStarted","Data":"e9fec3d0d723e3a77af483593eea917a88edc14a4e1df7d6accbf0ad282008bd"} Jul 24 15:46:19.958225 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:19.958156 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5zzlc" podStartSLOduration=34.113965636 podStartE2EDuration="35.958139104s" totalImagesPullingTime="1.844173468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:19.957440807 +0000 UTC m=+54.875411121" watchObservedRunningTime="2026-07-24 15:46:19.958139104 +0000 UTC m=+54.876109425" Jul 24 15:46:25.623271 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:25.623233 2569 scope.go:117] "RemoveContainer" containerID="93ceb84cce2a1961901a3a8da5ff51ec6c0119b2c6f168605541f3d2183ee0d8" Jul 24 15:46:25.803758 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:25.803737 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qm2n9" Jul 24 15:46:25.962711 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:25.961320 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 15:46:25.962711 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:25.961858 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/1.log" Jul 24 15:46:25.962711 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:25.961892 2569 generic.go:356] "Generic (PLEG): container finished" podID="70e5d390-4438-46a0-9e53-fae9badc534f" containerID="17edd1887d45a4d0276075b11222d6caf11e5b3afe966375cd856cdc7a6c64c4" exitCode=255 Jul 24 15:46:25.962711 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:25.961939 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" event={"ID":"70e5d390-4438-46a0-9e53-fae9badc534f","Type":"ContainerDied","Data":"17edd1887d45a4d0276075b11222d6caf11e5b3afe966375cd856cdc7a6c64c4"} Jul 24 15:46:25.962711 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:25.962019 2569 scope.go:117] "RemoveContainer" containerID="93ceb84cce2a1961901a3a8da5ff51ec6c0119b2c6f168605541f3d2183ee0d8" Jul 24 15:46:25.962711 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:25.962524 2569 scope.go:117] "RemoveContainer" containerID="17edd1887d45a4d0276075b11222d6caf11e5b3afe966375cd856cdc7a6c64c4" Jul 24 15:46:25.963131 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:25.962764 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-285sz_openshift-console-operator(70e5d390-4438-46a0-9e53-fae9badc534f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" podUID="70e5d390-4438-46a0-9e53-fae9badc534f" Jul 24 15:46:26.966118 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:26.966090 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 15:46:27.465710 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:27.465678 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:27.465897 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:27.465724 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:27.466331 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:27.466310 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:27.467993 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:27.467972 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxpsj\" (UID: \"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396\") " pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:27.550195 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:27.550163 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vfsm2\"" Jul 24 15:46:27.558197 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:27.558178 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hxpsj" Jul 24 15:46:27.689840 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:27.689805 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hxpsj"] Jul 24 15:46:27.692447 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:27.692419 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0972d7f9_5bcd_4ca5_8e8e_e23d04b7b396.slice/crio-858980b91bd5181f60c668c4f5c3481ea5690804dbcd3898b8c24db75ad14ab9 WatchSource:0}: Error finding container 858980b91bd5181f60c668c4f5c3481ea5690804dbcd3898b8c24db75ad14ab9: Status 404 returned error can't find the container with id 858980b91bd5181f60c668c4f5c3481ea5690804dbcd3898b8c24db75ad14ab9 Jul 24 15:46:27.970828 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:27.970793 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxpsj" event={"ID":"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396","Type":"ContainerStarted","Data":"858980b91bd5181f60c668c4f5c3481ea5690804dbcd3898b8c24db75ad14ab9"} Jul 24 15:46:31.201909 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:31.201872 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret\") pod \"global-pull-secret-syncer-lj6pt\" (UID: \"d1348e8a-07ee-4ce2-afa4-2d580edd8df2\") " pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:46:31.202363 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:31.202043 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs\") pod \"network-metrics-daemon-pdlxm\" (UID: \"2ff63e8f-2348-417f-a83d-8e72db071ae7\") " pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:46:31.204711 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:31.204688 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 15:46:31.204830 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:31.204688 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 15:46:31.215264 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:31.215179 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d1348e8a-07ee-4ce2-afa4-2d580edd8df2-original-pull-secret\") pod \"global-pull-secret-syncer-lj6pt\" (UID: \"d1348e8a-07ee-4ce2-afa4-2d580edd8df2\") " pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:46:31.215264 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:31.215248 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff63e8f-2348-417f-a83d-8e72db071ae7-metrics-certs\") pod \"network-metrics-daemon-pdlxm\" (UID: \"2ff63e8f-2348-417f-a83d-8e72db071ae7\") " pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:46:31.303008 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:31.302971 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd7f5\" (UniqueName: \"kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5\") pod \"network-check-target-tchwh\" (UID: \"2615ac1e-bb0a-4653-a612-f1406ea7d173\") " pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:46:31.305648 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:31.305619 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd7f5\" (UniqueName: \"kubernetes.io/projected/2615ac1e-bb0a-4653-a612-f1406ea7d173-kube-api-access-bd7f5\") pod \"network-check-target-tchwh\" (UID: \"2615ac1e-bb0a-4653-a612-f1406ea7d173\") " pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:46:31.350505 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:31.350466 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lj6pt" Jul 24 15:46:31.409228 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:31.409178 2569 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:31.409412 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:31.409277 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:31.409700 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:31.409681 2569 scope.go:117] "RemoveContainer" containerID="17edd1887d45a4d0276075b11222d6caf11e5b3afe966375cd856cdc7a6c64c4" Jul 24 15:46:31.409932 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:31.409902 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-285sz_openshift-console-operator(70e5d390-4438-46a0-9e53-fae9badc534f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" podUID="70e5d390-4438-46a0-9e53-fae9badc534f" Jul 24 15:46:31.445378 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:31.445352 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pqxfk\"" Jul 24 15:46:31.452711 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:31.452633 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cbp7q\"" Jul 24 15:46:31.453659 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:31.453635 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pdlxm" Jul 24 15:46:31.461632 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:31.461606 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:46:32.911860 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:32.911834 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pdlxm"] Jul 24 15:46:32.914092 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:32.914062 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff63e8f_2348_417f_a83d_8e72db071ae7.slice/crio-5773a35187d59999bfb128c727ea10af5da51888b9387c348a6ced63e0d07c8d WatchSource:0}: Error finding container 5773a35187d59999bfb128c727ea10af5da51888b9387c348a6ced63e0d07c8d: Status 404 returned error can't find the container with id 5773a35187d59999bfb128c727ea10af5da51888b9387c348a6ced63e0d07c8d Jul 24 15:46:32.919474 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:32.919452 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jp9sz\" (UID: \"61cb38d8-9f5e-495b-931b-7561758ce3a3\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" Jul 24 15:46:32.919579 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:32.919530 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:32.922032 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:32.922011 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/61cb38d8-9f5e-495b-931b-7561758ce3a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jp9sz\" (UID: \"61cb38d8-9f5e-495b-931b-7561758ce3a3\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" Jul 24 15:46:32.922859 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:32.922837 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls\") pod \"image-registry-6d75669b64-9vf6v\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:32.991099 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:32.991062 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxpsj" event={"ID":"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396","Type":"ContainerStarted","Data":"0a5c965a0a179ca65fa344dc79170358299650fb0f45a3cb863507ec3eba21e1"} Jul 24 15:46:32.992107 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:32.992085 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pdlxm" event={"ID":"2ff63e8f-2348-417f-a83d-8e72db071ae7","Type":"ContainerStarted","Data":"5773a35187d59999bfb128c727ea10af5da51888b9387c348a6ced63e0d07c8d"} Jul 24 15:46:33.020643 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.020565 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d44kg\" (UID: \"2f97027f-5262-48c0-acd9-61ccc7b95bbf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d44kg" Jul 24 15:46:33.022927 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.022909 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2f97027f-5262-48c0-acd9-61ccc7b95bbf-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-d44kg\" (UID: \"2f97027f-5262-48c0-acd9-61ccc7b95bbf\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-d44kg" Jul 24 15:46:33.068172 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.068148 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nsfqx\"" Jul 24 15:46:33.076287 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.076267 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:33.093085 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.093067 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ch4fh\"" Jul 24 15:46:33.101267 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.101249 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" Jul 24 15:46:33.121842 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.121771 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:33.121842 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.121808 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:33.121842 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.121836 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls\") pod \"dns-default-fwkmh\" (UID: \"20a41a97-5d30-42f9-8b18-138802f0f554\") " pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:33.122556 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.122490 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe32eca6-90f7-4d13-a58f-156b373a18f9-service-ca-bundle\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:33.124621 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.124596 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe32eca6-90f7-4d13-a58f-156b373a18f9-metrics-certs\") pod \"router-default-65b698c97-9vtdf\" (UID: \"fe32eca6-90f7-4d13-a58f-156b373a18f9\") " pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:33.125281 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.125257 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/20a41a97-5d30-42f9-8b18-138802f0f554-metrics-tls\") pod \"dns-default-fwkmh\" (UID: \"20a41a97-5d30-42f9-8b18-138802f0f554\") " pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:33.137087 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.137015 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tchwh"] Jul 24 15:46:33.138718 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.138456 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lj6pt"] Jul 24 15:46:33.141773 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:33.141738 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2615ac1e_bb0a_4653_a612_f1406ea7d173.slice/crio-c026fcd34b6c4f0e3f66cf8b9a5580e3011462782ac41b4cb2d2f8a2964b1a36 WatchSource:0}: Error finding container c026fcd34b6c4f0e3f66cf8b9a5580e3011462782ac41b4cb2d2f8a2964b1a36: Status 404 returned error can't find the container with id c026fcd34b6c4f0e3f66cf8b9a5580e3011462782ac41b4cb2d2f8a2964b1a36 Jul 24 15:46:33.142385 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:33.142351 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1348e8a_07ee_4ce2_afa4_2d580edd8df2.slice/crio-c7d22881393be3eb4bf01b6e056eac78843bbedc17768f6f4915a418d78773ca WatchSource:0}: Error finding container c7d22881393be3eb4bf01b6e056eac78843bbedc17768f6f4915a418d78773ca: Status 404 returned error can't find the container with id c7d22881393be3eb4bf01b6e056eac78843bbedc17768f6f4915a418d78773ca Jul 24 15:46:33.166078 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.165398 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-5ncdb\"" Jul 24 15:46:33.172783 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.172762 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-d44kg" Jul 24 15:46:33.220824 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.220771 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vwj8g\"" Jul 24 15:46:33.223013 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.222955 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d75669b64-9vf6v"] Jul 24 15:46:33.227055 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:33.227031 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6555d7aa_ea7e_4dbf_8bb1_be3e103afc39.slice/crio-2e370713f6df27eab30937cf1046053d9858f47bd37597102317afe7e6bf4115 WatchSource:0}: Error finding container 2e370713f6df27eab30937cf1046053d9858f47bd37597102317afe7e6bf4115: Status 404 returned error can't find the container with id 2e370713f6df27eab30937cf1046053d9858f47bd37597102317afe7e6bf4115 Jul 24 15:46:33.228123 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.228103 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:33.240813 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.240777 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz"] Jul 24 15:46:33.244647 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:33.244619 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61cb38d8_9f5e_495b_931b_7561758ce3a3.slice/crio-660378aafca63644df164bbf7620682c4b5509efe77ac84556d8c5f0171f940f WatchSource:0}: Error finding container 660378aafca63644df164bbf7620682c4b5509efe77ac84556d8c5f0171f940f: Status 404 returned error can't find the container with id 660378aafca63644df164bbf7620682c4b5509efe77ac84556d8c5f0171f940f Jul 24 15:46:33.294177 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.293951 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xjksm\"" Jul 24 15:46:33.303396 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.303331 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:33.328400 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.328377 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-d44kg"] Jul 24 15:46:33.333307 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:33.331527 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f97027f_5262_48c0_acd9_61ccc7b95bbf.slice/crio-eab816b196da94c10195b75683be22273fec06a05ec75f4ff900a68b577f167c WatchSource:0}: Error finding container eab816b196da94c10195b75683be22273fec06a05ec75f4ff900a68b577f167c: Status 404 returned error can't find the container with id eab816b196da94c10195b75683be22273fec06a05ec75f4ff900a68b577f167c Jul 24 15:46:33.402239 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.402116 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fwkmh"] Jul 24 15:46:33.408628 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:33.408563 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a41a97_5d30_42f9_8b18_138802f0f554.slice/crio-c09ee6aff6fae8bf5a43a354c3bb40c3238d2261ab042734fe55cdf8d6532266 WatchSource:0}: Error finding container c09ee6aff6fae8bf5a43a354c3bb40c3238d2261ab042734fe55cdf8d6532266: Status 404 returned error can't find the container with id c09ee6aff6fae8bf5a43a354c3bb40c3238d2261ab042734fe55cdf8d6532266 Jul 24 15:46:33.491229 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:33.491052 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-65b698c97-9vtdf"] Jul 24 15:46:33.496301 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:33.496241 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe32eca6_90f7_4d13_a58f_156b373a18f9.slice/crio-ff40f473e41e9ab67f14cb8c86f36bcef07ea2056602310cbe6b420add148508 WatchSource:0}: Error finding container ff40f473e41e9ab67f14cb8c86f36bcef07ea2056602310cbe6b420add148508: Status 404 returned error can't find the container with id ff40f473e41e9ab67f14cb8c86f36bcef07ea2056602310cbe6b420add148508 Jul 24 15:46:34.001016 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:34.000404 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" event={"ID":"61cb38d8-9f5e-495b-931b-7561758ce3a3","Type":"ContainerStarted","Data":"660378aafca63644df164bbf7620682c4b5509efe77ac84556d8c5f0171f940f"} Jul 24 15:46:34.007546 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:34.006458 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" event={"ID":"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39","Type":"ContainerStarted","Data":"f86a30967d7438fb05e5d3001926d022c740f6141d53f5c72acd1e19548f2f30"} Jul 24 15:46:34.007546 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:34.006547 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" event={"ID":"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39","Type":"ContainerStarted","Data":"2e370713f6df27eab30937cf1046053d9858f47bd37597102317afe7e6bf4115"} Jul 24 15:46:34.007546 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:34.007471 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:34.011435 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:34.010804 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tchwh" event={"ID":"2615ac1e-bb0a-4653-a612-f1406ea7d173","Type":"ContainerStarted","Data":"cdcc92d8560ae14e376294f48528286fde378a03f5ef0065c1c843888aa1ddbd"} Jul 24 15:46:34.011435 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:34.010848 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tchwh" event={"ID":"2615ac1e-bb0a-4653-a612-f1406ea7d173","Type":"ContainerStarted","Data":"c026fcd34b6c4f0e3f66cf8b9a5580e3011462782ac41b4cb2d2f8a2964b1a36"} Jul 24 15:46:34.011435 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:34.011363 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:46:34.016928 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:34.016839 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lj6pt" event={"ID":"d1348e8a-07ee-4ce2-afa4-2d580edd8df2","Type":"ContainerStarted","Data":"c7d22881393be3eb4bf01b6e056eac78843bbedc17768f6f4915a418d78773ca"} Jul 24 15:46:34.020760 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:34.020119 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-65b698c97-9vtdf" event={"ID":"fe32eca6-90f7-4d13-a58f-156b373a18f9","Type":"ContainerStarted","Data":"719fbd55db5bdd29aed7c066ad74d86fa82ecd3d9396782fb615b9f4a3c9d6d9"} Jul 24 15:46:34.020760 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:34.020167 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-65b698c97-9vtdf" event={"ID":"fe32eca6-90f7-4d13-a58f-156b373a18f9","Type":"ContainerStarted","Data":"ff40f473e41e9ab67f14cb8c86f36bcef07ea2056602310cbe6b420add148508"} Jul 24 15:46:34.024850 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:34.024772 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-d44kg" event={"ID":"2f97027f-5262-48c0-acd9-61ccc7b95bbf","Type":"ContainerStarted","Data":"eab816b196da94c10195b75683be22273fec06a05ec75f4ff900a68b577f167c"} Jul 24 15:46:34.033190 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:34.033126 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fwkmh" event={"ID":"20a41a97-5d30-42f9-8b18-138802f0f554","Type":"ContainerStarted","Data":"c09ee6aff6fae8bf5a43a354c3bb40c3238d2261ab042734fe55cdf8d6532266"} Jul 24 15:46:34.055313 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:34.054710 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" podStartSLOduration=73.054688843 podStartE2EDuration="1m13.054688843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:34.029642543 +0000 UTC m=+68.947612867" watchObservedRunningTime="2026-07-24 15:46:34.054688843 +0000 UTC m=+68.972659145" Jul 24 15:46:34.077944 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:34.076763 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tchwh" podStartSLOduration=69.076743479 podStartE2EDuration="1m9.076743479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:34.076032968 +0000 UTC m=+68.994003268" watchObservedRunningTime="2026-07-24 15:46:34.076743479 +0000 UTC m=+68.994713760" Jul 24 15:46:34.077944 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:34.077204 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-65b698c97-9vtdf" podStartSLOduration=50.077194767 podStartE2EDuration="50.077194767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:34.055168796 +0000 UTC m=+68.973139098" watchObservedRunningTime="2026-07-24 15:46:34.077194767 +0000 UTC m=+68.995165067" Jul 24 15:46:34.304715 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:34.304538 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:34.307782 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:34.307520 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:35.039423 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:35.038684 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:35.041671 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:35.041647 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-65b698c97-9vtdf" Jul 24 15:46:41.064778 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:41.064745 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxpsj" event={"ID":"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396","Type":"ContainerStarted","Data":"eae400d8f179eedf256b1e66e267e3b6b54dbe6214e734a9e3f0c8b60d8162cb"} Jul 24 15:46:41.066514 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:41.066482 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pdlxm" event={"ID":"2ff63e8f-2348-417f-a83d-8e72db071ae7","Type":"ContainerStarted","Data":"ea4239484d9c82d96cd26ddb995990763470097ade0a6190c0dc8798cbbfc2a9"} Jul 24 15:46:41.066638 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:41.066530 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pdlxm" event={"ID":"2ff63e8f-2348-417f-a83d-8e72db071ae7","Type":"ContainerStarted","Data":"d7ddb675669b3cc022e877d09ae7d57746720d92fff570b50f93c91978766e7b"} Jul 24 15:46:41.068051 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:41.068028 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-d44kg" event={"ID":"2f97027f-5262-48c0-acd9-61ccc7b95bbf","Type":"ContainerStarted","Data":"7b3ad8cdd4ca018e0e6e9ab245831cde2c09e84950160472f89d9a5de749f896"} Jul 24 15:46:41.070045 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:41.070025 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fwkmh" event={"ID":"20a41a97-5d30-42f9-8b18-138802f0f554","Type":"ContainerStarted","Data":"c7868709752c1691bcdb62d6943072b2c99c3ebbf5ed21d35ec82cb5afe20a4a"} Jul 24 15:46:41.070166 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:41.070064 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fwkmh" event={"ID":"20a41a97-5d30-42f9-8b18-138802f0f554","Type":"ContainerStarted","Data":"8d56cdd4b6418be649459ae9bfad43eceb3a3a3600dbe218de255624b8b7b7f5"} Jul 24 15:46:41.070166 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:41.070102 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:41.071591 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:41.071570 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" event={"ID":"61cb38d8-9f5e-495b-931b-7561758ce3a3","Type":"ContainerStarted","Data":"71342bd49cb1c2283abbdeca1fe2b7692a498638e392ead823372ec105d55934"} Jul 24 15:46:41.073048 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:41.073027 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lj6pt" event={"ID":"d1348e8a-07ee-4ce2-afa4-2d580edd8df2","Type":"ContainerStarted","Data":"4cffec6dae5d8e95cb3521993ab82481efacb51dddc1c5b0d8c35272fa98df67"} Jul 24 15:46:41.084548 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:41.084511 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pdlxm" podStartSLOduration=71.392294762 podStartE2EDuration="1m16.084499347s" totalImagesPullingTime="4.692204585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:41.08262224 +0000 UTC m=+76.000592542" watchObservedRunningTime="2026-07-24 15:46:41.084499347 +0000 UTC m=+76.002469703" Jul 24 15:46:41.099961 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:41.099924 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jp9sz" podStartSLOduration=50.284084805 podStartE2EDuration="57.099913395s" totalImagesPullingTime="6.81582859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:41.099798114 +0000 UTC m=+76.017768413" watchObservedRunningTime="2026-07-24 15:46:41.099913395 +0000 UTC m=+76.017883695" Jul 24 15:46:41.116675 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:41.116632 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fwkmh" podStartSLOduration=34.465400715 podStartE2EDuration="41.116618266s" totalImagesPullingTime="6.651217551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:41.116324673 +0000 UTC m=+76.034294974" watchObservedRunningTime="2026-07-24 15:46:41.116618266 +0000 UTC m=+76.034588565" Jul 24 15:46:41.134838 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:41.134793 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lj6pt" podStartSLOduration=69.215550239 podStartE2EDuration="1m16.134779407s" totalImagesPullingTime="6.919229168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:41.133828305 +0000 UTC m=+76.051798605" watchObservedRunningTime="2026-07-24 15:46:41.134779407 +0000 UTC m=+76.052749708" Jul 24 15:46:41.148936 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:41.148891 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-d44kg" podStartSLOduration=50.444478722 podStartE2EDuration="57.148876229s" totalImagesPullingTime="6.704397507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:41.148468467 +0000 UTC m=+76.066438770" watchObservedRunningTime="2026-07-24 15:46:41.148876229 +0000 UTC m=+76.066846529" Jul 24 15:46:41.622289 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:41.622257 2569 scope.go:117] "RemoveContainer" containerID="17edd1887d45a4d0276075b11222d6caf11e5b3afe966375cd856cdc7a6c64c4" Jul 24 15:46:41.622495 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:41.622471 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-285sz_openshift-console-operator(70e5d390-4438-46a0-9e53-fae9badc534f)\"" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" podUID="70e5d390-4438-46a0-9e53-fae9badc534f" Jul 24 15:46:45.095536 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:45.095494 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxpsj" event={"ID":"0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396","Type":"ContainerStarted","Data":"59213347dac74462878039515a38deafdc0f7faed1e68f66accf5a625ac430d6"} Jul 24 15:46:45.114411 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:45.114349 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hxpsj" podStartSLOduration=17.799689731 podStartE2EDuration="34.114334472s" totalImagesPullingTime="16.314644741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:46:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:45.114318083 +0000 UTC m=+80.032288382" watchObservedRunningTime="2026-07-24 15:46:45.114334472 +0000 UTC m=+80.032304772" Jul 24 15:46:51.084952 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.084818 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fwkmh" Jul 24 15:46:51.119068 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.119035 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p"] Jul 24 15:46:51.122770 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.122747 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" Jul 24 15:46:51.125445 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.125418 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 24 15:46:51.125445 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.125438 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 24 15:46:51.125640 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.125481 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mtk77\"" Jul 24 15:46:51.125640 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.125514 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 15:46:51.133395 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.133374 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p"] Jul 24 15:46:51.156270 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.156236 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf899\" (UniqueName: \"kubernetes.io/projected/4092a80d-7a41-487d-a985-8f8bee7528f0-kube-api-access-qf899\") pod \"openshift-state-metrics-b8f49f5c6-2k59p\" (UID: \"4092a80d-7a41-487d-a985-8f8bee7528f0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" Jul 24 15:46:51.156480 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.156293 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4092a80d-7a41-487d-a985-8f8bee7528f0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-2k59p\" (UID: \"4092a80d-7a41-487d-a985-8f8bee7528f0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" Jul 24 15:46:51.156480 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.156380 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4092a80d-7a41-487d-a985-8f8bee7528f0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-2k59p\" (UID: \"4092a80d-7a41-487d-a985-8f8bee7528f0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" Jul 24 15:46:51.156480 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.156426 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4092a80d-7a41-487d-a985-8f8bee7528f0-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-2k59p\" (UID: \"4092a80d-7a41-487d-a985-8f8bee7528f0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" Jul 24 15:46:51.158622 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.158599 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mfwls"] Jul 24 15:46:51.160692 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.160677 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.163513 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.163487 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 15:46:51.163632 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.163605 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 15:46:51.163692 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.163608 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7fcwn\"" Jul 24 15:46:51.163754 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.163739 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 15:46:51.257590 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.257552 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-node-exporter-textfile\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.257590 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.257589 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-node-exporter-accelerators-collector-config\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.257815 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.257688 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-root\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.257815 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.257726 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf899\" (UniqueName: \"kubernetes.io/projected/4092a80d-7a41-487d-a985-8f8bee7528f0-kube-api-access-qf899\") pod \"openshift-state-metrics-b8f49f5c6-2k59p\" (UID: \"4092a80d-7a41-487d-a985-8f8bee7528f0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" Jul 24 15:46:51.257815 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.257752 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cwxx\" (UniqueName: \"kubernetes.io/projected/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-kube-api-access-2cwxx\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.257815 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.257811 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4092a80d-7a41-487d-a985-8f8bee7528f0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-2k59p\" (UID: \"4092a80d-7a41-487d-a985-8f8bee7528f0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" Jul 24 15:46:51.258021 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.257831 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-sys\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.258021 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.257847 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-node-exporter-wtmp\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.258021 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.257866 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-node-exporter-tls\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.258021 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.257926 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4092a80d-7a41-487d-a985-8f8bee7528f0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-2k59p\" (UID: \"4092a80d-7a41-487d-a985-8f8bee7528f0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" Jul 24 15:46:51.258021 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.257964 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4092a80d-7a41-487d-a985-8f8bee7528f0-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-2k59p\" (UID: \"4092a80d-7a41-487d-a985-8f8bee7528f0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" Jul 24 15:46:51.258021 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.257985 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-metrics-client-ca\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.258021 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.258000 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.258320 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:51.258026 2569 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 24 15:46:51.258320 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:51.258112 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4092a80d-7a41-487d-a985-8f8bee7528f0-openshift-state-metrics-tls podName:4092a80d-7a41-487d-a985-8f8bee7528f0 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:51.758090595 +0000 UTC m=+86.676060873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/4092a80d-7a41-487d-a985-8f8bee7528f0-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-2k59p" (UID: "4092a80d-7a41-487d-a985-8f8bee7528f0") : secret "openshift-state-metrics-tls" not found Jul 24 15:46:51.258653 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.258637 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4092a80d-7a41-487d-a985-8f8bee7528f0-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-2k59p\" (UID: \"4092a80d-7a41-487d-a985-8f8bee7528f0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" Jul 24 15:46:51.260235 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.260202 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4092a80d-7a41-487d-a985-8f8bee7528f0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-2k59p\" (UID: \"4092a80d-7a41-487d-a985-8f8bee7528f0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" Jul 24 15:46:51.266943 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.266925 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf899\" (UniqueName: \"kubernetes.io/projected/4092a80d-7a41-487d-a985-8f8bee7528f0-kube-api-access-qf899\") pod \"openshift-state-metrics-b8f49f5c6-2k59p\" (UID: \"4092a80d-7a41-487d-a985-8f8bee7528f0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" Jul 24 15:46:51.359172 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.359093 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-node-exporter-textfile\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.359172 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.359124 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-node-exporter-accelerators-collector-config\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.359172 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.359164 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-root\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.359453 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.359180 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cwxx\" (UniqueName: \"kubernetes.io/projected/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-kube-api-access-2cwxx\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.359453 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.359198 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-sys\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.359453 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.359227 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-node-exporter-wtmp\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.359453 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.359247 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-node-exporter-tls\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.359453 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.359281 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-root\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.359453 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.359294 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-metrics-client-ca\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.359453 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.359309 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-sys\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.359453 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.359317 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.359453 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.359390 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-node-exporter-wtmp\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.359453 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:51.359412 2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 24 15:46:51.359453 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.359453 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-node-exporter-textfile\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.359999 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:46:51.359541 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-node-exporter-tls podName:8fbcb1ed-20ae-45aa-9406-141c2f9fddd0 nodeName:}" failed. No retries permitted until 2026-07-24 15:46:51.859510801 +0000 UTC m=+86.777481093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-node-exporter-tls") pod "node-exporter-mfwls" (UID: "8fbcb1ed-20ae-45aa-9406-141c2f9fddd0") : secret "node-exporter-tls" not found Jul 24 15:46:51.359999 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.359797 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-node-exporter-accelerators-collector-config\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.359999 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.359889 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-metrics-client-ca\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.361821 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.361798 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.370671 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.370645 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cwxx\" (UniqueName: \"kubernetes.io/projected/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-kube-api-access-2cwxx\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.761907 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.761865 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4092a80d-7a41-487d-a985-8f8bee7528f0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-2k59p\" (UID: \"4092a80d-7a41-487d-a985-8f8bee7528f0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" Jul 24 15:46:51.764313 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.764287 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4092a80d-7a41-487d-a985-8f8bee7528f0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-2k59p\" (UID: \"4092a80d-7a41-487d-a985-8f8bee7528f0\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" Jul 24 15:46:51.862331 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.862295 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-node-exporter-tls\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:51.864898 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:51.864870 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8fbcb1ed-20ae-45aa-9406-141c2f9fddd0-node-exporter-tls\") pod \"node-exporter-mfwls\" (UID: \"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0\") " pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:52.035884 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.035785 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" Jul 24 15:46:52.071009 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.070973 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mfwls" Jul 24 15:46:52.086032 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:52.085993 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbcb1ed_20ae_45aa_9406_141c2f9fddd0.slice/crio-3d02bb6f35276d199d38cca0b0405bbbc413b9b48677788b6149408ad0e37651 WatchSource:0}: Error finding container 3d02bb6f35276d199d38cca0b0405bbbc413b9b48677788b6149408ad0e37651: Status 404 returned error can't find the container with id 3d02bb6f35276d199d38cca0b0405bbbc413b9b48677788b6149408ad0e37651 Jul 24 15:46:52.126172 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.126137 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mfwls" event={"ID":"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0","Type":"ContainerStarted","Data":"3d02bb6f35276d199d38cca0b0405bbbc413b9b48677788b6149408ad0e37651"} Jul 24 15:46:52.180615 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.180562 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p"] Jul 24 15:46:52.182613 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:52.182582 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4092a80d_7a41_487d_a985_8f8bee7528f0.slice/crio-f8325fecc16e29c349037ded3666e718cf7235a4e8d4f1e62be47edf68a01708 WatchSource:0}: Error finding container f8325fecc16e29c349037ded3666e718cf7235a4e8d4f1e62be47edf68a01708: Status 404 returned error can't find the container with id f8325fecc16e29c349037ded3666e718cf7235a4e8d4f1e62be47edf68a01708 Jul 24 15:46:52.212644 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.212619 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:46:52.215503 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.215487 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.218016 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.217992 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 24 15:46:52.218146 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.218127 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 24 15:46:52.218198 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.218127 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ztczc\"" Jul 24 15:46:52.218318 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.218302 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 24 15:46:52.218379 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.218370 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 24 15:46:52.218469 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.218451 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 24 15:46:52.218524 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.218472 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 24 15:46:52.218576 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.218536 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 24 15:46:52.218870 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.218787 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 24 15:46:52.219298 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.219278 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 24 15:46:52.233357 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.233330 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:46:52.266286 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.266259 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a0ee7d3-cb07-4103-808b-778f78c3ee51-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.266406 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.266303 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.266406 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.266331 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a0ee7d3-cb07-4103-808b-778f78c3ee51-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.266406 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.266374 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjtcd\" (UniqueName: \"kubernetes.io/projected/5a0ee7d3-cb07-4103-808b-778f78c3ee51-kube-api-access-kjtcd\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.266653 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.266421 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.266653 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.266451 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a0ee7d3-cb07-4103-808b-778f78c3ee51-config-out\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.266653 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.266542 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a0ee7d3-cb07-4103-808b-778f78c3ee51-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.266653 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.266585 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.266653 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.266633 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.266864 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.266663 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a0ee7d3-cb07-4103-808b-778f78c3ee51-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.266864 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.266691 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.266864 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.266719 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.266864 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.266754 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-web-config\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.367509 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.367474 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.367674 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.367516 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.367674 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.367625 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-web-config\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.367674 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.367666 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a0ee7d3-cb07-4103-808b-778f78c3ee51-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.367893 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.367710 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.367893 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.367743 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a0ee7d3-cb07-4103-808b-778f78c3ee51-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.367893 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.367766 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjtcd\" (UniqueName: \"kubernetes.io/projected/5a0ee7d3-cb07-4103-808b-778f78c3ee51-kube-api-access-kjtcd\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.367893 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.367802 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.367893 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.367828 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a0ee7d3-cb07-4103-808b-778f78c3ee51-config-out\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.367893 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.367864 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a0ee7d3-cb07-4103-808b-778f78c3ee51-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.368174 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.367905 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.368174 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.367957 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.368174 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.367988 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a0ee7d3-cb07-4103-808b-778f78c3ee51-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.368174 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.368088 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a0ee7d3-cb07-4103-808b-778f78c3ee51-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.368666 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.368645 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a0ee7d3-cb07-4103-808b-778f78c3ee51-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.368760 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.368648 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a0ee7d3-cb07-4103-808b-778f78c3ee51-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.370992 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.370964 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.371773 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.371491 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a0ee7d3-cb07-4103-808b-778f78c3ee51-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.371773 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.371579 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.371773 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.371676 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a0ee7d3-cb07-4103-808b-778f78c3ee51-config-out\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.371971 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.371958 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.372103 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.372081 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-web-config\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.372145 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.372069 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.372611 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.372578 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-config-volume\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.373627 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.373606 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.375309 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.375293 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjtcd\" (UniqueName: \"kubernetes.io/projected/5a0ee7d3-cb07-4103-808b-778f78c3ee51-kube-api-access-kjtcd\") pod \"alertmanager-main-0\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.530984 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.530949 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:46:52.622727 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.622699 2569 scope.go:117] "RemoveContainer" containerID="17edd1887d45a4d0276075b11222d6caf11e5b3afe966375cd856cdc7a6c64c4" Jul 24 15:46:52.701592 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:52.701476 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:46:52.704592 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:52.704562 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0ee7d3_cb07_4103_808b_778f78c3ee51.slice/crio-da9d161577cf8e3a361a24330d810a5f00cd43ceea1344cbd2e4fc9869887162 WatchSource:0}: Error finding container da9d161577cf8e3a361a24330d810a5f00cd43ceea1344cbd2e4fc9869887162: Status 404 returned error can't find the container with id da9d161577cf8e3a361a24330d810a5f00cd43ceea1344cbd2e4fc9869887162 Jul 24 15:46:53.081369 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.081336 2569 patch_prober.go:28] interesting pod/image-registry-6d75669b64-9vf6v container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 24 15:46:53.081513 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.081397 2569 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" podUID="6555d7aa-ea7e-4dbf-8bb1-be3e103afc39" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 15:46:53.116741 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.116702 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-759b6bd54c-dxnbh"] Jul 24 15:46:53.120657 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.120636 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.123419 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.123149 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-cghls\"" Jul 24 15:46:53.123419 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.123398 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 24 15:46:53.123591 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.123415 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 24 15:46:53.123648 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.123612 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 24 15:46:53.123770 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.123754 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 24 15:46:53.123849 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.123840 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3n64t0gbu459a\"" Jul 24 15:46:53.123927 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.123909 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 24 15:46:53.132008 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.131984 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-759b6bd54c-dxnbh"] Jul 24 15:46:53.137420 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.137397 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 15:46:53.137506 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.137486 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" event={"ID":"70e5d390-4438-46a0-9e53-fae9badc534f","Type":"ContainerStarted","Data":"4271526f761462031070e459a81f79dfd998013e6e06e686c50d5f300e85c3e4"} Jul 24 15:46:53.138110 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.138063 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:53.139874 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.139802 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a0ee7d3-cb07-4103-808b-778f78c3ee51","Type":"ContainerStarted","Data":"da9d161577cf8e3a361a24330d810a5f00cd43ceea1344cbd2e4fc9869887162"} Jul 24 15:46:53.142837 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.142709 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" event={"ID":"4092a80d-7a41-487d-a985-8f8bee7528f0","Type":"ContainerStarted","Data":"cf87eea50478bcf7dccfd876fca5ee55449f159e6dec69f18306678cc9803c11"} Jul 24 15:46:53.142837 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.142767 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" event={"ID":"4092a80d-7a41-487d-a985-8f8bee7528f0","Type":"ContainerStarted","Data":"3a1747d4e6fb8c9b737db4b358ec1db7cfabaebceaeb9ec202c9017513c54bae"} Jul 24 15:46:53.142837 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.142797 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" event={"ID":"4092a80d-7a41-487d-a985-8f8bee7528f0","Type":"ContainerStarted","Data":"f8325fecc16e29c349037ded3666e718cf7235a4e8d4f1e62be47edf68a01708"} Jul 24 15:46:53.144645 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.144624 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mfwls" event={"ID":"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0","Type":"ContainerStarted","Data":"b303c66c191f2c8c31e9c2fa7287bc829f60207904cb321a22b7a891e4457a7e"} Jul 24 15:46:53.174837 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.174805 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e99f3198-7006-4f15-878b-9329f27bda15-secret-thanos-querier-tls\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.175085 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.175065 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e99f3198-7006-4f15-878b-9329f27bda15-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.175236 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.175182 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e99f3198-7006-4f15-878b-9329f27bda15-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.175332 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.175282 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e99f3198-7006-4f15-878b-9329f27bda15-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.175389 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.175344 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7qb4\" (UniqueName: \"kubernetes.io/projected/e99f3198-7006-4f15-878b-9329f27bda15-kube-api-access-b7qb4\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.176728 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.176701 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e99f3198-7006-4f15-878b-9329f27bda15-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.182002 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.181840 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e99f3198-7006-4f15-878b-9329f27bda15-metrics-client-ca\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.182002 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.181914 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e99f3198-7006-4f15-878b-9329f27bda15-secret-grpc-tls\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.198930 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.198876 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" podStartSLOduration=61.673601921 podStartE2EDuration="1m9.198854639s" totalImagesPullingTime="7.525252718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:53.197148098 +0000 UTC m=+88.115118399" watchObservedRunningTime="2026-07-24 15:46:53.198854639 +0000 UTC m=+88.116824940" Jul 24 15:46:53.282480 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.282391 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e99f3198-7006-4f15-878b-9329f27bda15-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.282480 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.282462 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7qb4\" (UniqueName: \"kubernetes.io/projected/e99f3198-7006-4f15-878b-9329f27bda15-kube-api-access-b7qb4\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.282696 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.282499 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e99f3198-7006-4f15-878b-9329f27bda15-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.282696 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.282532 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e99f3198-7006-4f15-878b-9329f27bda15-metrics-client-ca\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.282696 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.282565 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e99f3198-7006-4f15-878b-9329f27bda15-secret-grpc-tls\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.282696 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.282603 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e99f3198-7006-4f15-878b-9329f27bda15-secret-thanos-querier-tls\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.282696 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.282630 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e99f3198-7006-4f15-878b-9329f27bda15-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.282696 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.282674 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e99f3198-7006-4f15-878b-9329f27bda15-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.283607 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.283553 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e99f3198-7006-4f15-878b-9329f27bda15-metrics-client-ca\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.286001 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.285954 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e99f3198-7006-4f15-878b-9329f27bda15-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.286106 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.286079 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e99f3198-7006-4f15-878b-9329f27bda15-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.286434 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.286388 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e99f3198-7006-4f15-878b-9329f27bda15-secret-grpc-tls\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.286623 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.286598 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e99f3198-7006-4f15-878b-9329f27bda15-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.286907 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.286888 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e99f3198-7006-4f15-878b-9329f27bda15-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.287230 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.287187 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e99f3198-7006-4f15-878b-9329f27bda15-secret-thanos-querier-tls\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.292692 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.292670 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7qb4\" (UniqueName: \"kubernetes.io/projected/e99f3198-7006-4f15-878b-9329f27bda15-kube-api-access-b7qb4\") pod \"thanos-querier-759b6bd54c-dxnbh\" (UID: \"e99f3198-7006-4f15-878b-9329f27bda15\") " pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.432823 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.432799 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:53.582136 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.582102 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-759b6bd54c-dxnbh"] Jul 24 15:46:53.583562 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:53.583533 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99f3198_7006_4f15_878b_9329f27bda15.slice/crio-cf5f62c649638ffd3634849475a8b56a2a0f32509ae17da9a85a15680c2325df WatchSource:0}: Error finding container cf5f62c649638ffd3634849475a8b56a2a0f32509ae17da9a85a15680c2325df: Status 404 returned error can't find the container with id cf5f62c649638ffd3634849475a8b56a2a0f32509ae17da9a85a15680c2325df Jul 24 15:46:53.851900 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:53.851826 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-285sz" Jul 24 15:46:54.150002 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:54.149976 2569 generic.go:356] "Generic (PLEG): container finished" podID="8fbcb1ed-20ae-45aa-9406-141c2f9fddd0" containerID="b303c66c191f2c8c31e9c2fa7287bc829f60207904cb321a22b7a891e4457a7e" exitCode=0 Jul 24 15:46:54.150370 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:54.150077 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mfwls" event={"ID":"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0","Type":"ContainerDied","Data":"b303c66c191f2c8c31e9c2fa7287bc829f60207904cb321a22b7a891e4457a7e"} Jul 24 15:46:54.151387 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:54.151365 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" event={"ID":"e99f3198-7006-4f15-878b-9329f27bda15","Type":"ContainerStarted","Data":"cf5f62c649638ffd3634849475a8b56a2a0f32509ae17da9a85a15680c2325df"} Jul 24 15:46:54.154071 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:54.154010 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" event={"ID":"4092a80d-7a41-487d-a985-8f8bee7528f0","Type":"ContainerStarted","Data":"e3cf64c1eaae9893ffc10b6a0fd69502c573eed62d2b5908660f885fb2d5d909"} Jul 24 15:46:54.188104 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:54.187521 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-2k59p" podStartSLOduration=2.136912299 podStartE2EDuration="3.187504331s" totalImagesPullingTime="1.050592032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:54.186174341 +0000 UTC m=+89.104144643" watchObservedRunningTime="2026-07-24 15:46:54.187504331 +0000 UTC m=+89.105474631" Jul 24 15:46:55.161428 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:55.161387 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mfwls" event={"ID":"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0","Type":"ContainerStarted","Data":"ccec9f4ae8afa7a00512540b8cb727fce3a8f653bc6db3c63673acdc7bb60e51"} Jul 24 15:46:55.161865 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:55.161456 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mfwls" event={"ID":"8fbcb1ed-20ae-45aa-9406-141c2f9fddd0","Type":"ContainerStarted","Data":"f7b95c07f5a478ebec820c97f2accfa232eeabb257a90d10e3f73d69bd841644"} Jul 24 15:46:55.163145 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:55.163118 2569 generic.go:356] "Generic (PLEG): container finished" podID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerID="8fed4e42239921d2967491a14e7fb8bb9901fe38aef7bb7a3ed17bea562dc9f3" exitCode=0 Jul 24 15:46:55.163275 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:55.163199 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a0ee7d3-cb07-4103-808b-778f78c3ee51","Type":"ContainerDied","Data":"8fed4e42239921d2967491a14e7fb8bb9901fe38aef7bb7a3ed17bea562dc9f3"} Jul 24 15:46:55.184697 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:55.184651 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mfwls" podStartSLOduration=3.167745333 podStartE2EDuration="4.184633287s" totalImagesPullingTime="935.11195ms" totalInitContainerRuntime="81.776004ms" isStatefulPod=true podCreationTimestamp="2026-07-24 15:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:55.1821172 +0000 UTC m=+90.100087503" watchObservedRunningTime="2026-07-24 15:46:55.184633287 +0000 UTC m=+90.102603588" Jul 24 15:46:55.906941 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:55.906905 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-vhfgv"] Jul 24 15:46:55.913785 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:55.913763 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vhfgv" Jul 24 15:46:55.916427 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:55.916398 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 24 15:46:55.916427 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:55.916401 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mf6z9\"" Jul 24 15:46:55.919051 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:55.919029 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-vhfgv"] Jul 24 15:46:56.007795 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:56.007684 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cb84e7d4-45f5-490c-9ca2-a30a89a05a1e-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-vhfgv\" (UID: \"cb84e7d4-45f5-490c-9ca2-a30a89a05a1e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vhfgv" Jul 24 15:46:56.007973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:56.007828 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxr5z\" (UniqueName: \"kubernetes.io/projected/cb84e7d4-45f5-490c-9ca2-a30a89a05a1e-kube-api-access-cxr5z\") pod \"monitoring-plugin-68b7fc9574-vhfgv\" (UID: \"cb84e7d4-45f5-490c-9ca2-a30a89a05a1e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vhfgv" Jul 24 15:46:56.047845 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:56.047810 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:46:56.108997 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:56.108652 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxr5z\" (UniqueName: \"kubernetes.io/projected/cb84e7d4-45f5-490c-9ca2-a30a89a05a1e-kube-api-access-cxr5z\") pod \"monitoring-plugin-68b7fc9574-vhfgv\" (UID: \"cb84e7d4-45f5-490c-9ca2-a30a89a05a1e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vhfgv" Jul 24 15:46:56.108997 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:56.108754 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cb84e7d4-45f5-490c-9ca2-a30a89a05a1e-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-vhfgv\" (UID: \"cb84e7d4-45f5-490c-9ca2-a30a89a05a1e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vhfgv" Jul 24 15:46:56.111493 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:56.111468 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cb84e7d4-45f5-490c-9ca2-a30a89a05a1e-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-vhfgv\" (UID: \"cb84e7d4-45f5-490c-9ca2-a30a89a05a1e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vhfgv" Jul 24 15:46:56.117069 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:56.117043 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxr5z\" (UniqueName: \"kubernetes.io/projected/cb84e7d4-45f5-490c-9ca2-a30a89a05a1e-kube-api-access-cxr5z\") pod \"monitoring-plugin-68b7fc9574-vhfgv\" (UID: \"cb84e7d4-45f5-490c-9ca2-a30a89a05a1e\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vhfgv" Jul 24 15:46:56.170256 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:56.170141 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" event={"ID":"e99f3198-7006-4f15-878b-9329f27bda15","Type":"ContainerStarted","Data":"cc83a71591ac26295ec347081385231e14a442935e4cf5b9063cdf02cd38cd64"} Jul 24 15:46:56.170256 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:56.170252 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" event={"ID":"e99f3198-7006-4f15-878b-9329f27bda15","Type":"ContainerStarted","Data":"a5146022917c2b20fcccfdf32e2ebece6540f459faa0173e829b7422d9980bfc"} Jul 24 15:46:56.170752 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:56.170303 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" event={"ID":"e99f3198-7006-4f15-878b-9329f27bda15","Type":"ContainerStarted","Data":"8d4ca61f2c44699a3f801d5456936e21377405844e49d0a28a1488f34d70a0d0"} Jul 24 15:46:56.224960 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:56.224928 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vhfgv" Jul 24 15:46:56.360263 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:56.360238 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-vhfgv"] Jul 24 15:46:56.607982 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:46:56.607873 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb84e7d4_45f5_490c_9ca2_a30a89a05a1e.slice/crio-f7c7d19923d0be3a2d4f3e0e72dc6e5cddcb95219e26dcd92d74932d5743d6f0 WatchSource:0}: Error finding container f7c7d19923d0be3a2d4f3e0e72dc6e5cddcb95219e26dcd92d74932d5743d6f0: Status 404 returned error can't find the container with id f7c7d19923d0be3a2d4f3e0e72dc6e5cddcb95219e26dcd92d74932d5743d6f0 Jul 24 15:46:57.178460 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:57.178413 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a0ee7d3-cb07-4103-808b-778f78c3ee51","Type":"ContainerStarted","Data":"c4e5bcae0db8075ff0dbae43f6f593990eb5657ca576ecf5e7d3b38bb50196a2"} Jul 24 15:46:57.178831 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:57.178527 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a0ee7d3-cb07-4103-808b-778f78c3ee51","Type":"ContainerStarted","Data":"cde51174a199ee30e46ae43a967542efd71f18b7d9d6ad446a9b6596cc69e781"} Jul 24 15:46:57.178831 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:57.178568 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a0ee7d3-cb07-4103-808b-778f78c3ee51","Type":"ContainerStarted","Data":"5a0e8f5cb465d30f24ad4e4c831e042556e9ce3b2a28bf7e561dfb368571460f"} Jul 24 15:46:57.178831 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:57.178606 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a0ee7d3-cb07-4103-808b-778f78c3ee51","Type":"ContainerStarted","Data":"a302961facc0891a28f2da2ec787bbf68ad26354645f9a524375032f69234a90"} Jul 24 15:46:57.178831 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:57.178643 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a0ee7d3-cb07-4103-808b-778f78c3ee51","Type":"ContainerStarted","Data":"d5e472347120236a0a030cab2dcdec4c2c9f6d27ec5c39e657c7556acfc42bff"} Jul 24 15:46:57.178831 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:57.178677 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a0ee7d3-cb07-4103-808b-778f78c3ee51","Type":"ContainerStarted","Data":"8a194ad00213addbc65da304a9a8bca8dc058bd016382e427634e6ca0b728e5c"} Jul 24 15:46:57.179797 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:57.179769 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vhfgv" event={"ID":"cb84e7d4-45f5-490c-9ca2-a30a89a05a1e","Type":"ContainerStarted","Data":"f7c7d19923d0be3a2d4f3e0e72dc6e5cddcb95219e26dcd92d74932d5743d6f0"} Jul 24 15:46:57.183049 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:57.183022 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" event={"ID":"e99f3198-7006-4f15-878b-9329f27bda15","Type":"ContainerStarted","Data":"f2cef254f3512b983de31e867f2db902c383e569fd3a1b0ae0202f9201fde314"} Jul 24 15:46:57.183153 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:57.183097 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" event={"ID":"e99f3198-7006-4f15-878b-9329f27bda15","Type":"ContainerStarted","Data":"f3742631ad6db0abbaf862c25204e6a3f27b50d9dc4dfa5a981de420b0af5252"} Jul 24 15:46:57.183153 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:57.183139 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" event={"ID":"e99f3198-7006-4f15-878b-9329f27bda15","Type":"ContainerStarted","Data":"5d8773031f48a14506a53d31db5cebb2bde75b082762a517be7583db08668c86"} Jul 24 15:46:57.183288 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:57.183185 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:46:57.204664 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:57.204607 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.175805464 podStartE2EDuration="5.204588906s" totalImagesPullingTime="3.013277388s" totalInitContainerRuntime="15.506054ms" isStatefulPod=false podCreationTimestamp="2026-07-24 15:46:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:57.202896011 +0000 UTC m=+92.120866313" watchObservedRunningTime="2026-07-24 15:46:57.204588906 +0000 UTC m=+92.122559205" Jul 24 15:46:57.224428 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:57.224372 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" podStartSLOduration=1.244967869 podStartE2EDuration="4.224352153s" totalImagesPullingTime="2.979384284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:46:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:57.223762816 +0000 UTC m=+92.141733117" watchObservedRunningTime="2026-07-24 15:46:57.224352153 +0000 UTC m=+92.142322454" Jul 24 15:46:57.751707 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:57.751676 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d75669b64-9vf6v"] Jul 24 15:46:58.188176 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:58.188139 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vhfgv" event={"ID":"cb84e7d4-45f5-490c-9ca2-a30a89a05a1e","Type":"ContainerStarted","Data":"7e7b5b0e59a50e45090bf301e6009a0bf0914a41bb93b527050b9d0b99f666c8"} Jul 24 15:46:58.188644 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:58.188202 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vhfgv" Jul 24 15:46:58.193640 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:58.193620 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vhfgv" Jul 24 15:46:58.204088 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:46:58.204048 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vhfgv" podStartSLOduration=1.920295054 podStartE2EDuration="3.204032832s" totalImagesPullingTime="1.283737778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:46:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:58.203440428 +0000 UTC m=+93.121410729" watchObservedRunningTime="2026-07-24 15:46:58.204032832 +0000 UTC m=+93.122003131" Jul 24 15:47:03.194534 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:03.194504 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-759b6bd54c-dxnbh" Jul 24 15:47:06.045600 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:06.045568 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tchwh" Jul 24 15:47:10.728703 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:10.728668 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66d5655577-668vm"] Jul 24 15:47:10.951238 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:10.951180 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66d5655577-668vm"] Jul 24 15:47:10.951394 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:10.951317 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:10.954222 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:10.954178 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 15:47:10.954358 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:10.954248 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 15:47:10.955338 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:10.955319 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 15:47:10.955594 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:10.955580 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 15:47:10.955688 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:10.955673 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 15:47:10.956030 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:10.956017 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4nfqn\"" Jul 24 15:47:10.956030 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:10.956028 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 15:47:10.959854 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:10.959834 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 24 15:47:10.960875 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:10.960858 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 15:47:11.043811 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.043731 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-trusted-ca-bundle\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.043811 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.043767 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-console-serving-cert\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.043811 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.043791 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-console-oauth-config\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.044029 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.043862 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-console-config\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.044029 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.043907 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-service-ca\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.044029 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.043957 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-oauth-serving-cert\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.044029 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.044014 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmkgb\" (UniqueName: \"kubernetes.io/projected/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-kube-api-access-bmkgb\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.144797 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.144759 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-console-oauth-config\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.144797 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.144800 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-console-config\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.145031 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.144825 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-service-ca\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.145031 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.144860 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-oauth-serving-cert\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.145031 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.144891 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmkgb\" (UniqueName: \"kubernetes.io/projected/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-kube-api-access-bmkgb\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.145031 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.144918 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-trusted-ca-bundle\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.145031 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.144946 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-console-serving-cert\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.145713 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.145685 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-service-ca\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.145898 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.145866 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-oauth-serving-cert\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.145898 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.145759 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-console-config\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.146084 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.146022 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-trusted-ca-bundle\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.147519 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.147501 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-console-serving-cert\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.147609 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.147523 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-console-oauth-config\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.153503 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.153483 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmkgb\" (UniqueName: \"kubernetes.io/projected/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-kube-api-access-bmkgb\") pod \"console-66d5655577-668vm\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.261027 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.260997 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:11.420178 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:11.420147 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66d5655577-668vm"] Jul 24 15:47:11.423794 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:47:11.423757 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b931c58_1dc1_42b9_8ed4_5d48257ae89b.slice/crio-3a099dea31bb7c73c48b51f253e5e789173e9d030085c47baa13d2fc165a8e53 WatchSource:0}: Error finding container 3a099dea31bb7c73c48b51f253e5e789173e9d030085c47baa13d2fc165a8e53: Status 404 returned error can't find the container with id 3a099dea31bb7c73c48b51f253e5e789173e9d030085c47baa13d2fc165a8e53 Jul 24 15:47:12.251012 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:12.250973 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d5655577-668vm" event={"ID":"7b931c58-1dc1-42b9-8ed4-5d48257ae89b","Type":"ContainerStarted","Data":"3a099dea31bb7c73c48b51f253e5e789173e9d030085c47baa13d2fc165a8e53"} Jul 24 15:47:15.264633 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:15.264595 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d5655577-668vm" event={"ID":"7b931c58-1dc1-42b9-8ed4-5d48257ae89b","Type":"ContainerStarted","Data":"180947453aa859b27a611fcf3bb7b0cda554624bfadae9a29a3980958db83428"} Jul 24 15:47:15.284561 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:15.284515 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66d5655577-668vm" podStartSLOduration=2.029480427 podStartE2EDuration="5.284498329s" totalImagesPullingTime="3.255017902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:15.282279971 +0000 UTC m=+110.200250274" watchObservedRunningTime="2026-07-24 15:47:15.284498329 +0000 UTC m=+110.202468629" Jul 24 15:47:16.270532 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:16.270494 2569 generic.go:356] "Generic (PLEG): container finished" podID="2397af02-6f7f-4fde-905b-f733e7a77c47" containerID="4108fd5dbc6a39fef482b86f69fdde7c462372dda3c7ed7eb2787d96787ba56b" exitCode=0 Jul 24 15:47:16.270992 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:16.270564 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-zf2c2" event={"ID":"2397af02-6f7f-4fde-905b-f733e7a77c47","Type":"ContainerDied","Data":"4108fd5dbc6a39fef482b86f69fdde7c462372dda3c7ed7eb2787d96787ba56b"} Jul 24 15:47:16.270992 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:16.270916 2569 scope.go:117] "RemoveContainer" containerID="4108fd5dbc6a39fef482b86f69fdde7c462372dda3c7ed7eb2787d96787ba56b" Jul 24 15:47:17.276182 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:17.276148 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-zf2c2" event={"ID":"2397af02-6f7f-4fde-905b-f733e7a77c47","Type":"ContainerStarted","Data":"6b58f1d4dcdfb2e7bd5ec5c649baa438bf5be039b4b3d5302e477e29ad85e9b7"} Jul 24 15:47:17.967498 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:17.967472 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a0ee7d3-cb07-4103-808b-778f78c3ee51/init-config-reloader/0.log" Jul 24 15:47:18.167687 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:18.167652 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a0ee7d3-cb07-4103-808b-778f78c3ee51/alertmanager/0.log" Jul 24 15:47:18.371063 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:18.370986 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a0ee7d3-cb07-4103-808b-778f78c3ee51/config-reloader/0.log" Jul 24 15:47:18.568456 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:18.568428 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a0ee7d3-cb07-4103-808b-778f78c3ee51/kube-rbac-proxy-web/0.log" Jul 24 15:47:18.767928 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:18.767898 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a0ee7d3-cb07-4103-808b-778f78c3ee51/kube-rbac-proxy/0.log" Jul 24 15:47:18.967961 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:18.967934 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a0ee7d3-cb07-4103-808b-778f78c3ee51/kube-rbac-proxy-metric/0.log" Jul 24 15:47:19.168202 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:19.168176 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5a0ee7d3-cb07-4103-808b-778f78c3ee51/prom-label-proxy/0.log" Jul 24 15:47:19.368887 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:19.368866 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jp9sz_61cb38d8-9f5e-495b-931b-7561758ce3a3/cluster-monitoring-operator/0.log" Jul 24 15:47:20.367427 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:20.367400 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-vhfgv_cb84e7d4-45f5-490c-9ca2-a30a89a05a1e/monitoring-plugin/0.log" Jul 24 15:47:21.262204 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:21.262152 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:21.262437 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:21.262258 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:21.266975 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:21.266954 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:21.294025 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:21.293997 2569 generic.go:356] "Generic (PLEG): container finished" podID="ea54c847-2195-4b03-9e9d-31efaf46de3f" containerID="30e0a59abde673b50728e23ac1aa111dabb2979fbae4929b9e4f27b66c9711e5" exitCode=0 Jul 24 15:47:21.294183 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:21.294075 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m" event={"ID":"ea54c847-2195-4b03-9e9d-31efaf46de3f","Type":"ContainerDied","Data":"30e0a59abde673b50728e23ac1aa111dabb2979fbae4929b9e4f27b66c9711e5"} Jul 24 15:47:21.294438 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:21.294416 2569 scope.go:117] "RemoveContainer" containerID="30e0a59abde673b50728e23ac1aa111dabb2979fbae4929b9e4f27b66c9711e5" Jul 24 15:47:21.298273 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:21.298256 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66d5655577-668vm" Jul 24 15:47:21.767422 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:21.767394 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mfwls_8fbcb1ed-20ae-45aa-9406-141c2f9fddd0/init-textfile/0.log" Jul 24 15:47:21.967802 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:21.967777 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mfwls_8fbcb1ed-20ae-45aa-9406-141c2f9fddd0/node-exporter/0.log" Jul 24 15:47:22.168811 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:22.168782 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mfwls_8fbcb1ed-20ae-45aa-9406-141c2f9fddd0/kube-rbac-proxy/0.log" Jul 24 15:47:22.300959 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:22.300905 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m" event={"ID":"ea54c847-2195-4b03-9e9d-31efaf46de3f","Type":"ContainerStarted","Data":"a7cef38cdfa179d673b3591cd319fbb19a9ab127ebca68dea48648b1b572ed4a"} Jul 24 15:47:22.367971 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:22.367944 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-2k59p_4092a80d-7a41-487d-a985-8f8bee7528f0/kube-rbac-proxy-main/0.log" Jul 24 15:47:22.567825 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:22.567743 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-2k59p_4092a80d-7a41-487d-a985-8f8bee7528f0/kube-rbac-proxy-self/0.log" Jul 24 15:47:22.768133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:22.768104 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-2k59p_4092a80d-7a41-487d-a985-8f8bee7528f0/openshift-state-metrics/0.log" Jul 24 15:47:22.772657 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:22.772600 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" podUID="6555d7aa-ea7e-4dbf-8bb1-be3e103afc39" containerName="registry" containerID="cri-o://f86a30967d7438fb05e5d3001926d022c740f6141d53f5c72acd1e19548f2f30" gracePeriod=30 Jul 24 15:47:23.019505 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.019474 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:47:23.154299 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.154199 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls\") pod \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " Jul 24 15:47:23.154299 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.154250 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-trusted-ca\") pod \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " Jul 24 15:47:23.154528 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.154306 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-certificates\") pod \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " Jul 24 15:47:23.154528 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.154330 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jfjh\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-kube-api-access-6jfjh\") pod \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " Jul 24 15:47:23.154528 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.154467 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-bound-sa-token\") pod \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " Jul 24 15:47:23.154528 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.154509 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-ca-trust-extracted\") pod \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " Jul 24 15:47:23.154738 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.154548 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-installation-pull-secrets\") pod \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " Jul 24 15:47:23.154738 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.154574 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-image-registry-private-configuration\") pod \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\" (UID: \"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39\") " Jul 24 15:47:23.154738 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.154714 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39" (UID: "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:47:23.154884 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.154798 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39" (UID: "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:47:23.154940 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.154901 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-trusted-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:47:23.154940 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.154920 2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-certificates\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:47:23.157356 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.157333 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39" (UID: "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:47:23.157489 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.157472 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39" (UID: "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:47:23.157551 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.157486 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39" (UID: "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:47:23.157615 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.157559 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-kube-api-access-6jfjh" (OuterVolumeSpecName: "kube-api-access-6jfjh") pod "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39" (UID: "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39"). InnerVolumeSpecName "kube-api-access-6jfjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:47:23.157615 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.157598 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39" (UID: "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:47:23.163072 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.163044 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39" (UID: "6555d7aa-ea7e-4dbf-8bb1-be3e103afc39"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:47:23.255372 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.255340 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jfjh\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-kube-api-access-6jfjh\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:47:23.255372 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.255369 2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-bound-sa-token\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:47:23.255372 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.255379 2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-ca-trust-extracted\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:47:23.255604 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.255387 2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-installation-pull-secrets\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:47:23.255604 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.255397 2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-image-registry-private-configuration\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:47:23.255604 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.255407 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39-registry-tls\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:47:23.306274 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.306240 2569 generic.go:356] "Generic (PLEG): container finished" podID="6555d7aa-ea7e-4dbf-8bb1-be3e103afc39" containerID="f86a30967d7438fb05e5d3001926d022c740f6141d53f5c72acd1e19548f2f30" exitCode=0 Jul 24 15:47:23.306274 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.306273 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" Jul 24 15:47:23.306475 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.306302 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" event={"ID":"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39","Type":"ContainerDied","Data":"f86a30967d7438fb05e5d3001926d022c740f6141d53f5c72acd1e19548f2f30"} Jul 24 15:47:23.306475 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.306353 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d75669b64-9vf6v" event={"ID":"6555d7aa-ea7e-4dbf-8bb1-be3e103afc39","Type":"ContainerDied","Data":"2e370713f6df27eab30937cf1046053d9858f47bd37597102317afe7e6bf4115"} Jul 24 15:47:23.306475 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.306373 2569 scope.go:117] "RemoveContainer" containerID="f86a30967d7438fb05e5d3001926d022c740f6141d53f5c72acd1e19548f2f30" Jul 24 15:47:23.315381 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.315364 2569 scope.go:117] "RemoveContainer" containerID="f86a30967d7438fb05e5d3001926d022c740f6141d53f5c72acd1e19548f2f30" Jul 24 15:47:23.315658 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:47:23.315634 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f86a30967d7438fb05e5d3001926d022c740f6141d53f5c72acd1e19548f2f30\": container with ID starting with f86a30967d7438fb05e5d3001926d022c740f6141d53f5c72acd1e19548f2f30 not found: ID does not exist" containerID="f86a30967d7438fb05e5d3001926d022c740f6141d53f5c72acd1e19548f2f30" Jul 24 15:47:23.315741 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.315669 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f86a30967d7438fb05e5d3001926d022c740f6141d53f5c72acd1e19548f2f30"} err="failed to get container status \"f86a30967d7438fb05e5d3001926d022c740f6141d53f5c72acd1e19548f2f30\": rpc error: code = NotFound desc = could not find container \"f86a30967d7438fb05e5d3001926d022c740f6141d53f5c72acd1e19548f2f30\": container with ID starting with f86a30967d7438fb05e5d3001926d022c740f6141d53f5c72acd1e19548f2f30 not found: ID does not exist" Jul 24 15:47:23.331444 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.331418 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6d75669b64-9vf6v"] Jul 24 15:47:23.334636 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.334613 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6d75669b64-9vf6v"] Jul 24 15:47:23.626630 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:23.626553 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6555d7aa-ea7e-4dbf-8bb1-be3e103afc39" path="/var/lib/kubelet/pods/6555d7aa-ea7e-4dbf-8bb1-be3e103afc39/volumes" Jul 24 15:47:24.968411 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:24.968383 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759b6bd54c-dxnbh_e99f3198-7006-4f15-878b-9329f27bda15/thanos-query/0.log" Jul 24 15:47:25.171385 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:25.171356 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759b6bd54c-dxnbh_e99f3198-7006-4f15-878b-9329f27bda15/kube-rbac-proxy-web/0.log" Jul 24 15:47:25.367916 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:25.367837 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759b6bd54c-dxnbh_e99f3198-7006-4f15-878b-9329f27bda15/kube-rbac-proxy/0.log" Jul 24 15:47:25.568047 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:25.568019 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759b6bd54c-dxnbh_e99f3198-7006-4f15-878b-9329f27bda15/prom-label-proxy/0.log" Jul 24 15:47:25.768337 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:25.768304 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759b6bd54c-dxnbh_e99f3198-7006-4f15-878b-9329f27bda15/kube-rbac-proxy-rules/0.log" Jul 24 15:47:25.967615 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:25.967584 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759b6bd54c-dxnbh_e99f3198-7006-4f15-878b-9329f27bda15/kube-rbac-proxy-metrics/0.log" Jul 24 15:47:26.168030 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:26.167996 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-d44kg_2f97027f-5262-48c0-acd9-61ccc7b95bbf/networking-console-plugin/0.log" Jul 24 15:47:26.367308 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:26.367281 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 15:47:26.570305 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:26.570236 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/3.log" Jul 24 15:47:26.768803 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:26.768773 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-66d5655577-668vm_7b931c58-1dc1-42b9-8ed4-5d48257ae89b/console/0.log" Jul 24 15:47:27.369844 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:27.369821 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-65b698c97-9vtdf_fe32eca6-90f7-4d13-a58f-156b373a18f9/router/0.log" Jul 24 15:47:27.768980 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:27.768954 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6v2fh_d6d6c0cc-0d5f-407c-b9ea-00cf14c8076e/serve-healthcheck-canary/0.log" Jul 24 15:47:35.367054 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:35.367021 2569 generic.go:356] "Generic (PLEG): container finished" podID="8d07fcc0-3b9f-45f4-8e85-694a9ef1483f" containerID="d78575e0b45d4f1a6a95fd4988555cfb512ca38016c8370df69e781900f01229" exitCode=0 Jul 24 15:47:35.367481 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:35.367092 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wvx4n" event={"ID":"8d07fcc0-3b9f-45f4-8e85-694a9ef1483f","Type":"ContainerDied","Data":"d78575e0b45d4f1a6a95fd4988555cfb512ca38016c8370df69e781900f01229"} Jul 24 15:47:35.367481 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:35.367443 2569 scope.go:117] "RemoveContainer" containerID="d78575e0b45d4f1a6a95fd4988555cfb512ca38016c8370df69e781900f01229" Jul 24 15:47:36.373133 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:47:36.373099 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-wvx4n" event={"ID":"8d07fcc0-3b9f-45f4-8e85-694a9ef1483f","Type":"ContainerStarted","Data":"bb31196b8fd9ca0ea0b82180c81cca9a910f42cfce28dc8819db9d8e499e790d"} Jul 24 15:48:11.388240 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:11.388190 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:48:11.388731 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:11.388513 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="alertmanager" containerID="cri-o://8a194ad00213addbc65da304a9a8bca8dc058bd016382e427634e6ca0b728e5c" gracePeriod=120 Jul 24 15:48:11.388731 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:11.388573 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="kube-rbac-proxy-web" containerID="cri-o://a302961facc0891a28f2da2ec787bbf68ad26354645f9a524375032f69234a90" gracePeriod=120 Jul 24 15:48:11.388731 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:11.388618 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="kube-rbac-proxy" containerID="cri-o://5a0e8f5cb465d30f24ad4e4c831e042556e9ce3b2a28bf7e561dfb368571460f" gracePeriod=120 Jul 24 15:48:11.388731 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:11.388572 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="kube-rbac-proxy-metric" containerID="cri-o://cde51174a199ee30e46ae43a967542efd71f18b7d9d6ad446a9b6596cc69e781" gracePeriod=120 Jul 24 15:48:11.388956 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:11.388589 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="config-reloader" containerID="cri-o://d5e472347120236a0a030cab2dcdec4c2c9f6d27ec5c39e657c7556acfc42bff" gracePeriod=120 Jul 24 15:48:11.388956 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:11.388623 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="prom-label-proxy" containerID="cri-o://c4e5bcae0db8075ff0dbae43f6f593990eb5657ca576ecf5e7d3b38bb50196a2" gracePeriod=120 Jul 24 15:48:11.517395 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:11.517367 2569 generic.go:356] "Generic (PLEG): container finished" podID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerID="c4e5bcae0db8075ff0dbae43f6f593990eb5657ca576ecf5e7d3b38bb50196a2" exitCode=0 Jul 24 15:48:11.517395 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:11.517391 2569 generic.go:356] "Generic (PLEG): container finished" podID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerID="cde51174a199ee30e46ae43a967542efd71f18b7d9d6ad446a9b6596cc69e781" exitCode=0 Jul 24 15:48:11.517395 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:11.517397 2569 generic.go:356] "Generic (PLEG): container finished" podID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerID="5a0e8f5cb465d30f24ad4e4c831e042556e9ce3b2a28bf7e561dfb368571460f" exitCode=0 Jul 24 15:48:11.517573 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:11.517404 2569 generic.go:356] "Generic (PLEG): container finished" podID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerID="d5e472347120236a0a030cab2dcdec4c2c9f6d27ec5c39e657c7556acfc42bff" exitCode=0 Jul 24 15:48:11.517573 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:11.517409 2569 generic.go:356] "Generic (PLEG): container finished" podID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerID="8a194ad00213addbc65da304a9a8bca8dc058bd016382e427634e6ca0b728e5c" exitCode=0 Jul 24 15:48:11.517573 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:11.517435 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a0ee7d3-cb07-4103-808b-778f78c3ee51","Type":"ContainerDied","Data":"c4e5bcae0db8075ff0dbae43f6f593990eb5657ca576ecf5e7d3b38bb50196a2"} Jul 24 15:48:11.517573 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:11.517496 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a0ee7d3-cb07-4103-808b-778f78c3ee51","Type":"ContainerDied","Data":"cde51174a199ee30e46ae43a967542efd71f18b7d9d6ad446a9b6596cc69e781"} Jul 24 15:48:11.517573 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:11.517522 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a0ee7d3-cb07-4103-808b-778f78c3ee51","Type":"ContainerDied","Data":"5a0e8f5cb465d30f24ad4e4c831e042556e9ce3b2a28bf7e561dfb368571460f"} Jul 24 15:48:11.517573 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:11.517546 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a0ee7d3-cb07-4103-808b-778f78c3ee51","Type":"ContainerDied","Data":"d5e472347120236a0a030cab2dcdec4c2c9f6d27ec5c39e657c7556acfc42bff"} Jul 24 15:48:11.517573 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:11.517569 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a0ee7d3-cb07-4103-808b-778f78c3ee51","Type":"ContainerDied","Data":"8a194ad00213addbc65da304a9a8bca8dc058bd016382e427634e6ca0b728e5c"} Jul 24 15:48:12.525630 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.525598 2569 generic.go:356] "Generic (PLEG): container finished" podID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerID="a302961facc0891a28f2da2ec787bbf68ad26354645f9a524375032f69234a90" exitCode=0 Jul 24 15:48:12.525959 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.525675 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a0ee7d3-cb07-4103-808b-778f78c3ee51","Type":"ContainerDied","Data":"a302961facc0891a28f2da2ec787bbf68ad26354645f9a524375032f69234a90"} Jul 24 15:48:12.633681 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.633653 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:12.789552 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.789463 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjtcd\" (UniqueName: \"kubernetes.io/projected/5a0ee7d3-cb07-4103-808b-778f78c3ee51-kube-api-access-kjtcd\") pod \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " Jul 24 15:48:12.789552 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.789527 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a0ee7d3-cb07-4103-808b-778f78c3ee51-metrics-client-ca\") pod \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " Jul 24 15:48:12.789776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.789571 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a0ee7d3-cb07-4103-808b-778f78c3ee51-tls-assets\") pod \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " Jul 24 15:48:12.789776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.789603 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " Jul 24 15:48:12.789776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.789704 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a0ee7d3-cb07-4103-808b-778f78c3ee51-config-out\") pod \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " Jul 24 15:48:12.789776 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.789748 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-cluster-tls-config\") pod \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " Jul 24 15:48:12.789996 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.789784 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-kube-rbac-proxy-web\") pod \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " Jul 24 15:48:12.789996 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.789817 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a0ee7d3-cb07-4103-808b-778f78c3ee51-alertmanager-main-db\") pod \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " Jul 24 15:48:12.789996 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.789853 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-main-tls\") pod \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " Jul 24 15:48:12.789996 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.789883 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-web-config\") pod \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " Jul 24 15:48:12.789996 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.789912 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-kube-rbac-proxy\") pod \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " Jul 24 15:48:12.789996 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.789951 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-config-volume\") pod \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " Jul 24 15:48:12.789996 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.789984 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a0ee7d3-cb07-4103-808b-778f78c3ee51-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "5a0ee7d3-cb07-4103-808b-778f78c3ee51" (UID: "5a0ee7d3-cb07-4103-808b-778f78c3ee51"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:48:12.790379 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.789996 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a0ee7d3-cb07-4103-808b-778f78c3ee51-alertmanager-trusted-ca-bundle\") pod \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\" (UID: \"5a0ee7d3-cb07-4103-808b-778f78c3ee51\") " Jul 24 15:48:12.790379 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.790352 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a0ee7d3-cb07-4103-808b-778f78c3ee51-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "5a0ee7d3-cb07-4103-808b-778f78c3ee51" (UID: "5a0ee7d3-cb07-4103-808b-778f78c3ee51"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:48:12.790631 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.790593 2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a0ee7d3-cb07-4103-808b-778f78c3ee51-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:12.790705 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.790639 2569 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a0ee7d3-cb07-4103-808b-778f78c3ee51-metrics-client-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:12.791585 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.791299 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a0ee7d3-cb07-4103-808b-778f78c3ee51-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "5a0ee7d3-cb07-4103-808b-778f78c3ee51" (UID: "5a0ee7d3-cb07-4103-808b-778f78c3ee51"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:48:12.792534 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.792510 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0ee7d3-cb07-4103-808b-778f78c3ee51-kube-api-access-kjtcd" (OuterVolumeSpecName: "kube-api-access-kjtcd") pod "5a0ee7d3-cb07-4103-808b-778f78c3ee51" (UID: "5a0ee7d3-cb07-4103-808b-778f78c3ee51"). InnerVolumeSpecName "kube-api-access-kjtcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:48:12.792651 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.792573 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "5a0ee7d3-cb07-4103-808b-778f78c3ee51" (UID: "5a0ee7d3-cb07-4103-808b-778f78c3ee51"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:48:12.792963 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.792940 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0ee7d3-cb07-4103-808b-778f78c3ee51-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5a0ee7d3-cb07-4103-808b-778f78c3ee51" (UID: "5a0ee7d3-cb07-4103-808b-778f78c3ee51"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:48:12.793173 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.793133 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a0ee7d3-cb07-4103-808b-778f78c3ee51-config-out" (OuterVolumeSpecName: "config-out") pod "5a0ee7d3-cb07-4103-808b-778f78c3ee51" (UID: "5a0ee7d3-cb07-4103-808b-778f78c3ee51"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:48:12.793571 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.793535 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "5a0ee7d3-cb07-4103-808b-778f78c3ee51" (UID: "5a0ee7d3-cb07-4103-808b-778f78c3ee51"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:48:12.793677 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.793597 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "5a0ee7d3-cb07-4103-808b-778f78c3ee51" (UID: "5a0ee7d3-cb07-4103-808b-778f78c3ee51"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:48:12.794763 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.794744 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-config-volume" (OuterVolumeSpecName: "config-volume") pod "5a0ee7d3-cb07-4103-808b-778f78c3ee51" (UID: "5a0ee7d3-cb07-4103-808b-778f78c3ee51"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:48:12.794856 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.794769 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "5a0ee7d3-cb07-4103-808b-778f78c3ee51" (UID: "5a0ee7d3-cb07-4103-808b-778f78c3ee51"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:48:12.798049 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.798026 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "5a0ee7d3-cb07-4103-808b-778f78c3ee51" (UID: "5a0ee7d3-cb07-4103-808b-778f78c3ee51"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:48:12.803844 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.803823 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-web-config" (OuterVolumeSpecName: "web-config") pod "5a0ee7d3-cb07-4103-808b-778f78c3ee51" (UID: "5a0ee7d3-cb07-4103-808b-778f78c3ee51"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:48:12.891373 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.891323 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjtcd\" (UniqueName: \"kubernetes.io/projected/5a0ee7d3-cb07-4103-808b-778f78c3ee51-kube-api-access-kjtcd\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:12.891373 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.891367 2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a0ee7d3-cb07-4103-808b-778f78c3ee51-tls-assets\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:12.891373 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.891379 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:12.891373 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.891388 2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a0ee7d3-cb07-4103-808b-778f78c3ee51-config-out\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:12.891644 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.891396 2569 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-cluster-tls-config\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:12.891644 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.891407 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:12.891644 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.891419 2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5a0ee7d3-cb07-4103-808b-778f78c3ee51-alertmanager-main-db\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:12.891644 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.891428 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-main-tls\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:12.891644 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.891436 2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-web-config\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:12.891644 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.891445 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:12.891644 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:12.891452 2569 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5a0ee7d3-cb07-4103-808b-778f78c3ee51-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:13.536656 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.536614 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5a0ee7d3-cb07-4103-808b-778f78c3ee51","Type":"ContainerDied","Data":"da9d161577cf8e3a361a24330d810a5f00cd43ceea1344cbd2e4fc9869887162"} Jul 24 15:48:13.537058 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.536661 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.537058 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.536740 2569 scope.go:117] "RemoveContainer" containerID="c4e5bcae0db8075ff0dbae43f6f593990eb5657ca576ecf5e7d3b38bb50196a2" Jul 24 15:48:13.545334 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.545314 2569 scope.go:117] "RemoveContainer" containerID="cde51174a199ee30e46ae43a967542efd71f18b7d9d6ad446a9b6596cc69e781" Jul 24 15:48:13.552262 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.552246 2569 scope.go:117] "RemoveContainer" containerID="5a0e8f5cb465d30f24ad4e4c831e042556e9ce3b2a28bf7e561dfb368571460f" Jul 24 15:48:13.558827 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.558806 2569 scope.go:117] "RemoveContainer" containerID="a302961facc0891a28f2da2ec787bbf68ad26354645f9a524375032f69234a90" Jul 24 15:48:13.561091 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.561071 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:48:13.565707 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.565685 2569 scope.go:117] "RemoveContainer" containerID="d5e472347120236a0a030cab2dcdec4c2c9f6d27ec5c39e657c7556acfc42bff" Jul 24 15:48:13.568288 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.568269 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:48:13.572802 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.572785 2569 scope.go:117] "RemoveContainer" containerID="8a194ad00213addbc65da304a9a8bca8dc058bd016382e427634e6ca0b728e5c" Jul 24 15:48:13.579606 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.579588 2569 scope.go:117] "RemoveContainer" containerID="8fed4e42239921d2967491a14e7fb8bb9901fe38aef7bb7a3ed17bea562dc9f3" Jul 24 15:48:13.596259 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596237 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:48:13.596530 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596519 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="prom-label-proxy" Jul 24 15:48:13.596577 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596532 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="prom-label-proxy" Jul 24 15:48:13.596577 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596545 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6555d7aa-ea7e-4dbf-8bb1-be3e103afc39" containerName="registry" Jul 24 15:48:13.596577 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596550 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6555d7aa-ea7e-4dbf-8bb1-be3e103afc39" containerName="registry" Jul 24 15:48:13.596577 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596558 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="kube-rbac-proxy-web" Jul 24 15:48:13.596577 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596564 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="kube-rbac-proxy-web" Jul 24 15:48:13.596577 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596575 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="alertmanager" Jul 24 15:48:13.596577 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596580 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="alertmanager" Jul 24 15:48:13.596796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596588 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="init-config-reloader" Jul 24 15:48:13.596796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596594 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="init-config-reloader" Jul 24 15:48:13.596796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596606 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="kube-rbac-proxy" Jul 24 15:48:13.596796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596611 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="kube-rbac-proxy" Jul 24 15:48:13.596796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596618 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="config-reloader" Jul 24 15:48:13.596796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596623 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="config-reloader" Jul 24 15:48:13.596796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596631 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="kube-rbac-proxy-metric" Jul 24 15:48:13.596796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596636 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="kube-rbac-proxy-metric" Jul 24 15:48:13.596796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596682 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="kube-rbac-proxy" Jul 24 15:48:13.596796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596689 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="alertmanager" Jul 24 15:48:13.596796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596696 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="prom-label-proxy" Jul 24 15:48:13.596796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596701 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="kube-rbac-proxy-web" Jul 24 15:48:13.596796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596708 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="config-reloader" Jul 24 15:48:13.596796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596714 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6555d7aa-ea7e-4dbf-8bb1-be3e103afc39" containerName="registry" Jul 24 15:48:13.596796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.596721 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" containerName="kube-rbac-proxy-metric" Jul 24 15:48:13.601840 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.601826 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.604376 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.604356 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 24 15:48:13.605081 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.605062 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 24 15:48:13.605567 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.605547 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ztczc\"" Jul 24 15:48:13.605567 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.605565 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 24 15:48:13.605723 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.605584 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 24 15:48:13.605723 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.605585 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 24 15:48:13.605723 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.605680 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 24 15:48:13.605932 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.605739 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 24 15:48:13.605932 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.605825 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 24 15:48:13.611633 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.611616 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 24 15:48:13.627248 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.627224 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a0ee7d3-cb07-4103-808b-778f78c3ee51" path="/var/lib/kubelet/pods/5a0ee7d3-cb07-4103-808b-778f78c3ee51/volumes" Jul 24 15:48:13.627701 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.627688 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:48:13.698232 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.698174 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-config-volume\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.698232 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.698235 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.698446 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.698255 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65hs8\" (UniqueName: \"kubernetes.io/projected/9df211ef-c1bc-4f57-9959-8ba32bc884d5-kube-api-access-65hs8\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.698446 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.698292 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.698446 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.698311 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df211ef-c1bc-4f57-9959-8ba32bc884d5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.698446 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.698333 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9df211ef-c1bc-4f57-9959-8ba32bc884d5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.698446 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.698367 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9df211ef-c1bc-4f57-9959-8ba32bc884d5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.698597 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.698449 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.698597 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.698482 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9df211ef-c1bc-4f57-9959-8ba32bc884d5-config-out\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.698597 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.698535 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.698696 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.698590 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9df211ef-c1bc-4f57-9959-8ba32bc884d5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.698696 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.698636 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-web-config\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.698696 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.698654 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.799330 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.799255 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-web-config\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.799330 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.799288 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.799330 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.799314 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-config-volume\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.799568 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.799333 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.799568 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.799440 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65hs8\" (UniqueName: \"kubernetes.io/projected/9df211ef-c1bc-4f57-9959-8ba32bc884d5-kube-api-access-65hs8\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.799568 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.799514 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.799568 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.799541 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df211ef-c1bc-4f57-9959-8ba32bc884d5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.799762 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.799572 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9df211ef-c1bc-4f57-9959-8ba32bc884d5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.799762 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.799599 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9df211ef-c1bc-4f57-9959-8ba32bc884d5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.799762 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.799654 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.799762 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.799699 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9df211ef-c1bc-4f57-9959-8ba32bc884d5-config-out\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.799762 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.799726 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.800084 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.799768 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9df211ef-c1bc-4f57-9959-8ba32bc884d5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.800793 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.800463 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9df211ef-c1bc-4f57-9959-8ba32bc884d5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.800910 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.800832 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df211ef-c1bc-4f57-9959-8ba32bc884d5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.802193 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.802166 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-web-config\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.802306 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.802266 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.802371 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.802323 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.802583 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.802544 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9df211ef-c1bc-4f57-9959-8ba32bc884d5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.802824 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.802799 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-config-volume\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.802927 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.802867 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9df211ef-c1bc-4f57-9959-8ba32bc884d5-config-out\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.803131 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.803114 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9df211ef-c1bc-4f57-9959-8ba32bc884d5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.803400 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.803382 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.803542 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.803525 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.804314 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.804292 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9df211ef-c1bc-4f57-9959-8ba32bc884d5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.807613 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.807595 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65hs8\" (UniqueName: \"kubernetes.io/projected/9df211ef-c1bc-4f57-9959-8ba32bc884d5-kube-api-access-65hs8\") pod \"alertmanager-main-0\" (UID: \"9df211ef-c1bc-4f57-9959-8ba32bc884d5\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:13.910799 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:13.910768 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:48:14.045505 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:14.045476 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:48:14.048291 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:48:14.048264 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df211ef_c1bc_4f57_9959_8ba32bc884d5.slice/crio-45b13b8e8dfc216e489230fd34edfff9edbba919f7be35358f66f7ae63b78738 WatchSource:0}: Error finding container 45b13b8e8dfc216e489230fd34edfff9edbba919f7be35358f66f7ae63b78738: Status 404 returned error can't find the container with id 45b13b8e8dfc216e489230fd34edfff9edbba919f7be35358f66f7ae63b78738 Jul 24 15:48:14.542763 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:14.542727 2569 generic.go:356] "Generic (PLEG): container finished" podID="9df211ef-c1bc-4f57-9959-8ba32bc884d5" containerID="a6ee8e113667a0f7fab410e67176a9f13a26162c36000f083afbecc986a1eacc" exitCode=0 Jul 24 15:48:14.543158 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:14.542808 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df211ef-c1bc-4f57-9959-8ba32bc884d5","Type":"ContainerDied","Data":"a6ee8e113667a0f7fab410e67176a9f13a26162c36000f083afbecc986a1eacc"} Jul 24 15:48:14.543158 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:14.542872 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df211ef-c1bc-4f57-9959-8ba32bc884d5","Type":"ContainerStarted","Data":"45b13b8e8dfc216e489230fd34edfff9edbba919f7be35358f66f7ae63b78738"} Jul 24 15:48:15.413534 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.413498 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-d7966675b-28qzj"] Jul 24 15:48:15.416971 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.416953 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.421831 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.421807 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 24 15:48:15.421965 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.421809 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 24 15:48:15.421965 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.421908 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 24 15:48:15.422394 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.422377 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 24 15:48:15.422545 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.422527 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8skp2\"" Jul 24 15:48:15.433332 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.433308 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 24 15:48:15.435815 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.435793 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 24 15:48:15.452671 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.452643 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d7966675b-28qzj"] Jul 24 15:48:15.515772 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.515744 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e62d39b7-bb9b-46f9-9106-eb880de55932-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.515932 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.515779 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e62d39b7-bb9b-46f9-9106-eb880de55932-serving-certs-ca-bundle\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.515932 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.515807 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e62d39b7-bb9b-46f9-9106-eb880de55932-secret-telemeter-client\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.515932 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.515844 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e62d39b7-bb9b-46f9-9106-eb880de55932-metrics-client-ca\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.515932 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.515887 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngtfm\" (UniqueName: \"kubernetes.io/projected/e62d39b7-bb9b-46f9-9106-eb880de55932-kube-api-access-ngtfm\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.515932 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.515908 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e62d39b7-bb9b-46f9-9106-eb880de55932-federate-client-tls\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.516085 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.515958 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e62d39b7-bb9b-46f9-9106-eb880de55932-telemeter-client-tls\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.516085 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.515994 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e62d39b7-bb9b-46f9-9106-eb880de55932-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.550455 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.550420 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df211ef-c1bc-4f57-9959-8ba32bc884d5","Type":"ContainerStarted","Data":"0116cff5f2d5f90263d80a7416325e7727308901b60871c07f98dfc596a5fbe3"} Jul 24 15:48:15.550825 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.550490 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df211ef-c1bc-4f57-9959-8ba32bc884d5","Type":"ContainerStarted","Data":"b967012714ebbb39b7e1d21f880df4b760b8355b726fdecc71e352fff04c28e7"} Jul 24 15:48:15.550825 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.550525 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df211ef-c1bc-4f57-9959-8ba32bc884d5","Type":"ContainerStarted","Data":"23f3e6ead084e4db266573fa2c59b2063d8f13422ff195806d6423383cf8288f"} Jul 24 15:48:15.550825 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.550549 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df211ef-c1bc-4f57-9959-8ba32bc884d5","Type":"ContainerStarted","Data":"b7a84768b8191d40c24825446acff7cbeb0f924e0f9d4c054255255063fdff2d"} Jul 24 15:48:15.550825 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.550580 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df211ef-c1bc-4f57-9959-8ba32bc884d5","Type":"ContainerStarted","Data":"4bdc4505cf4046af0a3d15dc59ab6cf115b640338a0742c81814b9472b41ea55"} Jul 24 15:48:15.550825 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.550604 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df211ef-c1bc-4f57-9959-8ba32bc884d5","Type":"ContainerStarted","Data":"c08b7b29de8c39e370f02ab7a57ea24f92d09322a139b2c4c078e6a01a67e2b7"} Jul 24 15:48:15.576930 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.576881 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.562776833 podStartE2EDuration="2.576863895s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.087062ms" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:15.574820629 +0000 UTC m=+170.492790930" watchObservedRunningTime="2026-07-24 15:48:15.576863895 +0000 UTC m=+170.494834250" Jul 24 15:48:15.616573 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.616536 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e62d39b7-bb9b-46f9-9106-eb880de55932-telemeter-client-tls\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.616764 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.616589 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e62d39b7-bb9b-46f9-9106-eb880de55932-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.616764 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.616610 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e62d39b7-bb9b-46f9-9106-eb880de55932-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.616764 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.616633 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e62d39b7-bb9b-46f9-9106-eb880de55932-serving-certs-ca-bundle\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.616764 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.616663 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e62d39b7-bb9b-46f9-9106-eb880de55932-secret-telemeter-client\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.616764 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.616701 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e62d39b7-bb9b-46f9-9106-eb880de55932-metrics-client-ca\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.616764 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.616735 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngtfm\" (UniqueName: \"kubernetes.io/projected/e62d39b7-bb9b-46f9-9106-eb880de55932-kube-api-access-ngtfm\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.617085 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.616777 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e62d39b7-bb9b-46f9-9106-eb880de55932-federate-client-tls\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.617530 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.617503 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e62d39b7-bb9b-46f9-9106-eb880de55932-serving-certs-ca-bundle\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.617702 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.617678 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e62d39b7-bb9b-46f9-9106-eb880de55932-telemeter-trusted-ca-bundle\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.618147 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.618122 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e62d39b7-bb9b-46f9-9106-eb880de55932-metrics-client-ca\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.619345 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.619322 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e62d39b7-bb9b-46f9-9106-eb880de55932-secret-telemeter-client\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.619503 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.619483 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e62d39b7-bb9b-46f9-9106-eb880de55932-telemeter-client-tls\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.619601 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.619577 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e62d39b7-bb9b-46f9-9106-eb880de55932-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.620346 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.620325 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e62d39b7-bb9b-46f9-9106-eb880de55932-federate-client-tls\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.625877 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.625849 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngtfm\" (UniqueName: \"kubernetes.io/projected/e62d39b7-bb9b-46f9-9106-eb880de55932-kube-api-access-ngtfm\") pod \"telemeter-client-d7966675b-28qzj\" (UID: \"e62d39b7-bb9b-46f9-9106-eb880de55932\") " pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.727107 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.727027 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" Jul 24 15:48:15.861755 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:15.861732 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-d7966675b-28qzj"] Jul 24 15:48:15.863634 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:48:15.863611 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62d39b7_bb9b_46f9_9106_eb880de55932.slice/crio-14e11e9ac6734c8d6faebd7986979f6eee0e3cd1e221b79df9a3070db35b5cbe WatchSource:0}: Error finding container 14e11e9ac6734c8d6faebd7986979f6eee0e3cd1e221b79df9a3070db35b5cbe: Status 404 returned error can't find the container with id 14e11e9ac6734c8d6faebd7986979f6eee0e3cd1e221b79df9a3070db35b5cbe Jul 24 15:48:16.561192 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:16.561150 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" event={"ID":"e62d39b7-bb9b-46f9-9106-eb880de55932","Type":"ContainerStarted","Data":"14e11e9ac6734c8d6faebd7986979f6eee0e3cd1e221b79df9a3070db35b5cbe"} Jul 24 15:48:18.573172 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:18.573135 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" event={"ID":"e62d39b7-bb9b-46f9-9106-eb880de55932","Type":"ContainerStarted","Data":"4dc3b37e79a99dd65a0d2f06084f2265b178d3ff33f818b01497f986da643c7e"} Jul 24 15:48:18.573540 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:18.573206 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" event={"ID":"e62d39b7-bb9b-46f9-9106-eb880de55932","Type":"ContainerStarted","Data":"bc337243f950a6c41a48e041baeb13fa8ccdfcfbc30048164693219474dd02a5"} Jul 24 15:48:18.573540 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:18.573255 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" event={"ID":"e62d39b7-bb9b-46f9-9106-eb880de55932","Type":"ContainerStarted","Data":"8b5b886c05fab155992ff047b9150c594fd525f62f36bf4ea29da168428359a0"} Jul 24 15:48:18.597857 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:18.597256 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-d7966675b-28qzj" podStartSLOduration=1.79545791 podStartE2EDuration="3.597236764s" totalImagesPullingTime="1.801778854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:18.595391666 +0000 UTC m=+173.513361965" watchObservedRunningTime="2026-07-24 15:48:18.597236764 +0000 UTC m=+173.515207061" Jul 24 15:48:30.175544 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:30.175510 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66d5655577-668vm"] Jul 24 15:48:55.640187 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:55.640129 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66d5655577-668vm" podUID="7b931c58-1dc1-42b9-8ed4-5d48257ae89b" containerName="console" containerID="cri-o://180947453aa859b27a611fcf3bb7b0cda554624bfadae9a29a3980958db83428" gracePeriod=15 Jul 24 15:48:55.894295 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:55.894236 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d5655577-668vm" Jul 24 15:48:56.083331 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.083294 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-console-config\") pod \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " Jul 24 15:48:56.083331 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.083335 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-console-oauth-config\") pod \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " Jul 24 15:48:56.083576 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.083366 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-oauth-serving-cert\") pod \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " Jul 24 15:48:56.083576 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.083404 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-trusted-ca-bundle\") pod \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " Jul 24 15:48:56.083576 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.083466 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-service-ca\") pod \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " Jul 24 15:48:56.083576 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.083503 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmkgb\" (UniqueName: \"kubernetes.io/projected/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-kube-api-access-bmkgb\") pod \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " Jul 24 15:48:56.083576 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.083554 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-console-serving-cert\") pod \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\" (UID: \"7b931c58-1dc1-42b9-8ed4-5d48257ae89b\") " Jul 24 15:48:56.083849 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.083824 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-console-config" (OuterVolumeSpecName: "console-config") pod "7b931c58-1dc1-42b9-8ed4-5d48257ae89b" (UID: "7b931c58-1dc1-42b9-8ed4-5d48257ae89b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:48:56.083889 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.083851 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7b931c58-1dc1-42b9-8ed4-5d48257ae89b" (UID: "7b931c58-1dc1-42b9-8ed4-5d48257ae89b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:48:56.083991 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.083966 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7b931c58-1dc1-42b9-8ed4-5d48257ae89b" (UID: "7b931c58-1dc1-42b9-8ed4-5d48257ae89b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:48:56.084186 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.084167 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-service-ca" (OuterVolumeSpecName: "service-ca") pod "7b931c58-1dc1-42b9-8ed4-5d48257ae89b" (UID: "7b931c58-1dc1-42b9-8ed4-5d48257ae89b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:48:56.085690 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.085667 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7b931c58-1dc1-42b9-8ed4-5d48257ae89b" (UID: "7b931c58-1dc1-42b9-8ed4-5d48257ae89b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:48:56.085786 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.085717 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7b931c58-1dc1-42b9-8ed4-5d48257ae89b" (UID: "7b931c58-1dc1-42b9-8ed4-5d48257ae89b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:48:56.085824 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.085773 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-kube-api-access-bmkgb" (OuterVolumeSpecName: "kube-api-access-bmkgb") pod "7b931c58-1dc1-42b9-8ed4-5d48257ae89b" (UID: "7b931c58-1dc1-42b9-8ed4-5d48257ae89b"). InnerVolumeSpecName "kube-api-access-bmkgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:48:56.184861 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.184772 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-service-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:56.184861 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.184806 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmkgb\" (UniqueName: \"kubernetes.io/projected/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-kube-api-access-bmkgb\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:56.184861 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.184817 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-console-serving-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:56.184861 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.184825 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-console-config\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:56.184861 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.184835 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-console-oauth-config\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:56.184861 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.184842 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-oauth-serving-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:56.184861 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.184850 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b931c58-1dc1-42b9-8ed4-5d48257ae89b-trusted-ca-bundle\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:48:56.726739 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.726706 2569 generic.go:356] "Generic (PLEG): container finished" podID="7b931c58-1dc1-42b9-8ed4-5d48257ae89b" containerID="180947453aa859b27a611fcf3bb7b0cda554624bfadae9a29a3980958db83428" exitCode=0 Jul 24 15:48:56.727096 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.726754 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66d5655577-668vm" Jul 24 15:48:56.727096 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.726799 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d5655577-668vm" event={"ID":"7b931c58-1dc1-42b9-8ed4-5d48257ae89b","Type":"ContainerDied","Data":"180947453aa859b27a611fcf3bb7b0cda554624bfadae9a29a3980958db83428"} Jul 24 15:48:56.727096 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.726854 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66d5655577-668vm" event={"ID":"7b931c58-1dc1-42b9-8ed4-5d48257ae89b","Type":"ContainerDied","Data":"3a099dea31bb7c73c48b51f253e5e789173e9d030085c47baa13d2fc165a8e53"} Jul 24 15:48:56.727096 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.726879 2569 scope.go:117] "RemoveContainer" containerID="180947453aa859b27a611fcf3bb7b0cda554624bfadae9a29a3980958db83428" Jul 24 15:48:56.736497 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.736480 2569 scope.go:117] "RemoveContainer" containerID="180947453aa859b27a611fcf3bb7b0cda554624bfadae9a29a3980958db83428" Jul 24 15:48:56.736736 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:48:56.736716 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"180947453aa859b27a611fcf3bb7b0cda554624bfadae9a29a3980958db83428\": container with ID starting with 180947453aa859b27a611fcf3bb7b0cda554624bfadae9a29a3980958db83428 not found: ID does not exist" containerID="180947453aa859b27a611fcf3bb7b0cda554624bfadae9a29a3980958db83428" Jul 24 15:48:56.736784 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.736745 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"180947453aa859b27a611fcf3bb7b0cda554624bfadae9a29a3980958db83428"} err="failed to get container status \"180947453aa859b27a611fcf3bb7b0cda554624bfadae9a29a3980958db83428\": rpc error: code = NotFound desc = could not find container \"180947453aa859b27a611fcf3bb7b0cda554624bfadae9a29a3980958db83428\": container with ID starting with 180947453aa859b27a611fcf3bb7b0cda554624bfadae9a29a3980958db83428 not found: ID does not exist" Jul 24 15:48:56.748766 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.748730 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66d5655577-668vm"] Jul 24 15:48:56.755832 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:56.755807 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66d5655577-668vm"] Jul 24 15:48:57.626477 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:48:57.626428 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b931c58-1dc1-42b9-8ed4-5d48257ae89b" path="/var/lib/kubelet/pods/7b931c58-1dc1-42b9-8ed4-5d48257ae89b/volumes" Jul 24 15:49:31.309750 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.309716 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56b965b5cd-zqgsx"] Jul 24 15:49:31.310356 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.310100 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b931c58-1dc1-42b9-8ed4-5d48257ae89b" containerName="console" Jul 24 15:49:31.310356 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.310112 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b931c58-1dc1-42b9-8ed4-5d48257ae89b" containerName="console" Jul 24 15:49:31.310356 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.310185 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b931c58-1dc1-42b9-8ed4-5d48257ae89b" containerName="console" Jul 24 15:49:31.314780 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.314761 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.317332 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.317312 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 15:49:31.317462 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.317374 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 15:49:31.317619 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.317602 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 15:49:31.317701 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.317643 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 15:49:31.317701 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.317671 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 15:49:31.318011 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.317995 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 15:49:31.318161 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.318149 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 15:49:31.318279 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.318259 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4nfqn\"" Jul 24 15:49:31.324943 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.324912 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56b965b5cd-zqgsx"] Jul 24 15:49:31.325094 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.325065 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 24 15:49:31.380926 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.380887 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-service-ca\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.381095 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.380932 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-trusted-ca-bundle\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.381095 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.380994 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b348b56-c220-40b8-907b-da9b58719243-console-oauth-config\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.381095 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.381045 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv76n\" (UniqueName: \"kubernetes.io/projected/6b348b56-c220-40b8-907b-da9b58719243-kube-api-access-mv76n\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.381095 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.381078 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-console-config\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.381249 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.381101 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b348b56-c220-40b8-907b-da9b58719243-console-serving-cert\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.381249 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.381182 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-oauth-serving-cert\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.481989 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.481955 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-oauth-serving-cert\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.482175 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.482003 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-service-ca\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.482175 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.482125 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-trusted-ca-bundle\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.482311 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.482181 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b348b56-c220-40b8-907b-da9b58719243-console-oauth-config\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.482311 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.482236 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv76n\" (UniqueName: \"kubernetes.io/projected/6b348b56-c220-40b8-907b-da9b58719243-kube-api-access-mv76n\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.482311 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.482269 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-console-config\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.482311 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.482291 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b348b56-c220-40b8-907b-da9b58719243-console-serving-cert\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.482772 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.482746 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-oauth-serving-cert\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.482886 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.482838 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-service-ca\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.482985 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.482962 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-console-config\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.483131 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.483117 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-trusted-ca-bundle\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.484657 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.484635 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b348b56-c220-40b8-907b-da9b58719243-console-oauth-config\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.484778 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.484757 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b348b56-c220-40b8-907b-da9b58719243-console-serving-cert\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.490194 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.490168 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv76n\" (UniqueName: \"kubernetes.io/projected/6b348b56-c220-40b8-907b-da9b58719243-kube-api-access-mv76n\") pod \"console-56b965b5cd-zqgsx\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.627300 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.627202 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:31.769852 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.769825 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56b965b5cd-zqgsx"] Jul 24 15:49:31.772632 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:49:31.772604 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b348b56_c220_40b8_907b_da9b58719243.slice/crio-f61d3c5300456c7dbf740ea5750cf0ea844ebe449c90455d66579cf1ab1c7b66 WatchSource:0}: Error finding container f61d3c5300456c7dbf740ea5750cf0ea844ebe449c90455d66579cf1ab1c7b66: Status 404 returned error can't find the container with id f61d3c5300456c7dbf740ea5750cf0ea844ebe449c90455d66579cf1ab1c7b66 Jul 24 15:49:31.868896 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.868835 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b965b5cd-zqgsx" event={"ID":"6b348b56-c220-40b8-907b-da9b58719243","Type":"ContainerStarted","Data":"fc5a9ba969ff38a3a2f40ccc0f91c46c5b0d6190f21aa05279e8a11836363675"} Jul 24 15:49:31.869095 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.868913 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b965b5cd-zqgsx" event={"ID":"6b348b56-c220-40b8-907b-da9b58719243","Type":"ContainerStarted","Data":"f61d3c5300456c7dbf740ea5750cf0ea844ebe449c90455d66579cf1ab1c7b66"} Jul 24 15:49:31.900653 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:31.900598 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56b965b5cd-zqgsx" podStartSLOduration=0.900579617 podStartE2EDuration="900.579617ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:49:31.899530566 +0000 UTC m=+246.817500905" watchObservedRunningTime="2026-07-24 15:49:31.900579617 +0000 UTC m=+246.818549917" Jul 24 15:49:41.628111 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:41.628078 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:41.628615 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:41.628392 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:41.632885 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:41.632867 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:49:41.913362 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:49:41.913332 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:50:25.424171 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:25.424143 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 15:50:25.428911 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:25.428882 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 15:50:25.430345 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:25.430323 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 15:50:25.435519 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:25.435498 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 15:50:25.435702 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:25.435683 2569 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 15:50:27.692120 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:27.692085 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr"] Jul 24 15:50:27.695370 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:27.695340 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr" Jul 24 15:50:27.697973 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:27.697945 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 15:50:27.698096 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:27.697973 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 15:50:27.699017 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:27.699000 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jnzpn\"" Jul 24 15:50:27.704080 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:27.704062 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr"] Jul 24 15:50:27.876631 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:27.876591 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr\" (UID: \"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr" Jul 24 15:50:27.876631 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:27.876635 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnr8r\" (UniqueName: \"kubernetes.io/projected/b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3-kube-api-access-qnr8r\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr\" (UID: \"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr" Jul 24 15:50:27.876884 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:27.876730 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr\" (UID: \"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr" Jul 24 15:50:27.977684 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:27.977596 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr\" (UID: \"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr" Jul 24 15:50:27.977684 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:27.977659 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr\" (UID: \"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr" Jul 24 15:50:27.977863 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:27.977753 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnr8r\" (UniqueName: \"kubernetes.io/projected/b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3-kube-api-access-qnr8r\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr\" (UID: \"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr" Jul 24 15:50:27.977983 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:27.977964 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr\" (UID: \"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr" Jul 24 15:50:27.978020 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:27.977994 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr\" (UID: \"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr" Jul 24 15:50:27.985779 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:27.985746 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnr8r\" (UniqueName: \"kubernetes.io/projected/b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3-kube-api-access-qnr8r\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr\" (UID: \"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr" Jul 24 15:50:28.004682 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:28.004652 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr" Jul 24 15:50:28.135122 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:28.135089 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr"] Jul 24 15:50:28.137271 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:50:28.137242 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f6efa3_ff9d_48c3_bcc6_fd44efac6af3.slice/crio-ddb150b9fa83faf100bce8731d1b6faea7ec0fa1fbf7de55565c3f7feaceeb06 WatchSource:0}: Error finding container ddb150b9fa83faf100bce8731d1b6faea7ec0fa1fbf7de55565c3f7feaceeb06: Status 404 returned error can't find the container with id ddb150b9fa83faf100bce8731d1b6faea7ec0fa1fbf7de55565c3f7feaceeb06 Jul 24 15:50:28.139287 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:28.139269 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:50:29.110286 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:29.110248 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr" event={"ID":"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3","Type":"ContainerStarted","Data":"ddb150b9fa83faf100bce8731d1b6faea7ec0fa1fbf7de55565c3f7feaceeb06"} Jul 24 15:50:35.141346 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:35.141311 2569 generic.go:356] "Generic (PLEG): container finished" podID="b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3" containerID="0acc4c6867988607321fd50d80ef4f4685b96400a5c4b038843c71c54bec86c9" exitCode=0 Jul 24 15:50:35.141804 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:35.141408 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr" event={"ID":"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3","Type":"ContainerDied","Data":"0acc4c6867988607321fd50d80ef4f4685b96400a5c4b038843c71c54bec86c9"} Jul 24 15:50:38.157034 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:38.156994 2569 generic.go:356] "Generic (PLEG): container finished" podID="b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3" containerID="f71865bc41327a9522ae65e9cf9a8ad9297715aea9d6b18499f14cf01e96bedc" exitCode=0 Jul 24 15:50:38.157469 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:38.157049 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr" event={"ID":"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3","Type":"ContainerDied","Data":"f71865bc41327a9522ae65e9cf9a8ad9297715aea9d6b18499f14cf01e96bedc"} Jul 24 15:50:46.197416 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:46.197376 2569 generic.go:356] "Generic (PLEG): container finished" podID="b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3" containerID="47122d236a4288cc8788d72d320bd5c89ecabc227c60c2c494cca451e1ddc804" exitCode=0 Jul 24 15:50:46.197828 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:46.197466 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr" event={"ID":"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3","Type":"ContainerDied","Data":"47122d236a4288cc8788d72d320bd5c89ecabc227c60c2c494cca451e1ddc804"} Jul 24 15:50:47.332023 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:47.331997 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr" Jul 24 15:50:47.455406 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:47.455322 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3-util\") pod \"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3\" (UID: \"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3\") " Jul 24 15:50:47.455406 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:47.455370 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3-bundle\") pod \"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3\" (UID: \"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3\") " Jul 24 15:50:47.455583 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:47.455448 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnr8r\" (UniqueName: \"kubernetes.io/projected/b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3-kube-api-access-qnr8r\") pod \"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3\" (UID: \"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3\") " Jul 24 15:50:47.456103 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:47.456075 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3-bundle" (OuterVolumeSpecName: "bundle") pod "b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3" (UID: "b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:50:47.457627 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:47.457607 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3-kube-api-access-qnr8r" (OuterVolumeSpecName: "kube-api-access-qnr8r") pod "b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3" (UID: "b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3"). InnerVolumeSpecName "kube-api-access-qnr8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:50:47.459505 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:47.459482 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3-util" (OuterVolumeSpecName: "util") pod "b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3" (UID: "b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:50:47.556670 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:47.556627 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnr8r\" (UniqueName: \"kubernetes.io/projected/b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3-kube-api-access-qnr8r\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:50:47.556670 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:47.556661 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3-util\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:50:47.556670 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:47.556672 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3-bundle\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:50:48.207541 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:48.207507 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr" event={"ID":"b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3","Type":"ContainerDied","Data":"ddb150b9fa83faf100bce8731d1b6faea7ec0fa1fbf7de55565c3f7feaceeb06"} Jul 24 15:50:48.207709 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:48.207559 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddb150b9fa83faf100bce8731d1b6faea7ec0fa1fbf7de55565c3f7feaceeb06" Jul 24 15:50:48.207709 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:48.207518 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408fbhr" Jul 24 15:50:55.721444 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:55.721411 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5b7km"] Jul 24 15:50:55.721872 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:55.721848 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3" containerName="pull" Jul 24 15:50:55.721872 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:55.721871 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3" containerName="pull" Jul 24 15:50:55.722023 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:55.721883 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3" containerName="extract" Jul 24 15:50:55.722023 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:55.721888 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3" containerName="extract" Jul 24 15:50:55.722023 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:55.721907 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3" containerName="util" Jul 24 15:50:55.722023 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:55.721914 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3" containerName="util" Jul 24 15:50:55.722023 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:55.721977 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f6efa3-ff9d-48c3-bcc6-fd44efac6af3" containerName="extract" Jul 24 15:50:55.725020 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:55.725005 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5b7km" Jul 24 15:50:55.727682 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:55.727663 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-wgl94\"" Jul 24 15:50:55.727784 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:55.727689 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:50:55.728035 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:55.728021 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 24 15:50:55.740194 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:55.740170 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5b7km"] Jul 24 15:50:55.830663 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:55.830626 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2baab308-9029-4ba6-814e-5d48f6ba1de5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-5b7km\" (UID: \"2baab308-9029-4ba6-814e-5d48f6ba1de5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5b7km" Jul 24 15:50:55.830829 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:55.830682 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftlp5\" (UniqueName: \"kubernetes.io/projected/2baab308-9029-4ba6-814e-5d48f6ba1de5-kube-api-access-ftlp5\") pod \"cert-manager-operator-controller-manager-55c444bf86-5b7km\" (UID: \"2baab308-9029-4ba6-814e-5d48f6ba1de5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5b7km" Jul 24 15:50:55.931754 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:55.931722 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2baab308-9029-4ba6-814e-5d48f6ba1de5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-5b7km\" (UID: \"2baab308-9029-4ba6-814e-5d48f6ba1de5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5b7km" Jul 24 15:50:55.931987 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:55.931764 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftlp5\" (UniqueName: \"kubernetes.io/projected/2baab308-9029-4ba6-814e-5d48f6ba1de5-kube-api-access-ftlp5\") pod \"cert-manager-operator-controller-manager-55c444bf86-5b7km\" (UID: \"2baab308-9029-4ba6-814e-5d48f6ba1de5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5b7km" Jul 24 15:50:55.932120 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:55.932100 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2baab308-9029-4ba6-814e-5d48f6ba1de5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-5b7km\" (UID: \"2baab308-9029-4ba6-814e-5d48f6ba1de5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5b7km" Jul 24 15:50:55.940962 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:55.940933 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftlp5\" (UniqueName: \"kubernetes.io/projected/2baab308-9029-4ba6-814e-5d48f6ba1de5-kube-api-access-ftlp5\") pod \"cert-manager-operator-controller-manager-55c444bf86-5b7km\" (UID: \"2baab308-9029-4ba6-814e-5d48f6ba1de5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5b7km" Jul 24 15:50:56.035118 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:56.035012 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5b7km" Jul 24 15:50:56.167843 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:56.167819 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5b7km"] Jul 24 15:50:56.169985 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:50:56.169958 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2baab308_9029_4ba6_814e_5d48f6ba1de5.slice/crio-ef5be6037d8da1f1303d0edb0f43bf42aff7f8eb1039ed733d9e6cb584fe14c3 WatchSource:0}: Error finding container ef5be6037d8da1f1303d0edb0f43bf42aff7f8eb1039ed733d9e6cb584fe14c3: Status 404 returned error can't find the container with id ef5be6037d8da1f1303d0edb0f43bf42aff7f8eb1039ed733d9e6cb584fe14c3 Jul 24 15:50:56.242444 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:56.242408 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5b7km" event={"ID":"2baab308-9029-4ba6-814e-5d48f6ba1de5","Type":"ContainerStarted","Data":"ef5be6037d8da1f1303d0edb0f43bf42aff7f8eb1039ed733d9e6cb584fe14c3"} Jul 24 15:50:58.255801 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:58.255759 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5b7km" event={"ID":"2baab308-9029-4ba6-814e-5d48f6ba1de5","Type":"ContainerStarted","Data":"20e1a8224eab2b0f19f72efbab7f30db1445633130ab6cb9012f268af8bec758"} Jul 24 15:50:58.280339 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:58.280278 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-5b7km" podStartSLOduration=1.623389293 podStartE2EDuration="3.280257163s" totalImagesPullingTime="1.65686787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:50:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:50:58.278908489 +0000 UTC m=+333.196878792" watchObservedRunningTime="2026-07-24 15:50:58.280257163 +0000 UTC m=+333.198227466" Jul 24 15:50:59.390429 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:59.390396 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng"] Jul 24 15:50:59.393499 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:59.393477 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng" Jul 24 15:50:59.396103 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:59.396081 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jnzpn\"" Jul 24 15:50:59.396194 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:59.396121 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 15:50:59.397385 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:59.397369 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 15:50:59.405343 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:59.405317 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng"] Jul 24 15:50:59.463927 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:59.463893 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckqjn\" (UniqueName: \"kubernetes.io/projected/6172bc03-a7dd-4306-8d35-aa844d32ed66-kube-api-access-ckqjn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng\" (UID: \"6172bc03-a7dd-4306-8d35-aa844d32ed66\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng" Jul 24 15:50:59.464120 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:59.463950 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6172bc03-a7dd-4306-8d35-aa844d32ed66-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng\" (UID: \"6172bc03-a7dd-4306-8d35-aa844d32ed66\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng" Jul 24 15:50:59.464120 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:59.464022 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6172bc03-a7dd-4306-8d35-aa844d32ed66-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng\" (UID: \"6172bc03-a7dd-4306-8d35-aa844d32ed66\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng" Jul 24 15:50:59.564598 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:59.564568 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckqjn\" (UniqueName: \"kubernetes.io/projected/6172bc03-a7dd-4306-8d35-aa844d32ed66-kube-api-access-ckqjn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng\" (UID: \"6172bc03-a7dd-4306-8d35-aa844d32ed66\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng" Jul 24 15:50:59.564788 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:59.564609 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6172bc03-a7dd-4306-8d35-aa844d32ed66-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng\" (UID: \"6172bc03-a7dd-4306-8d35-aa844d32ed66\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng" Jul 24 15:50:59.564788 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:59.564635 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6172bc03-a7dd-4306-8d35-aa844d32ed66-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng\" (UID: \"6172bc03-a7dd-4306-8d35-aa844d32ed66\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng" Jul 24 15:50:59.564987 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:59.564970 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6172bc03-a7dd-4306-8d35-aa844d32ed66-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng\" (UID: \"6172bc03-a7dd-4306-8d35-aa844d32ed66\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng" Jul 24 15:50:59.565058 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:59.565020 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6172bc03-a7dd-4306-8d35-aa844d32ed66-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng\" (UID: \"6172bc03-a7dd-4306-8d35-aa844d32ed66\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng" Jul 24 15:50:59.572968 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:59.572941 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckqjn\" (UniqueName: \"kubernetes.io/projected/6172bc03-a7dd-4306-8d35-aa844d32ed66-kube-api-access-ckqjn\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng\" (UID: \"6172bc03-a7dd-4306-8d35-aa844d32ed66\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng" Jul 24 15:50:59.702905 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:59.702827 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng" Jul 24 15:50:59.834709 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:50:59.834688 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng"] Jul 24 15:50:59.836833 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:50:59.836805 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6172bc03_a7dd_4306_8d35_aa844d32ed66.slice/crio-01407d33ed7dece9f1e5fca58db2137df1918fa5de268189132ab2692674fca4 WatchSource:0}: Error finding container 01407d33ed7dece9f1e5fca58db2137df1918fa5de268189132ab2692674fca4: Status 404 returned error can't find the container with id 01407d33ed7dece9f1e5fca58db2137df1918fa5de268189132ab2692674fca4 Jul 24 15:51:00.266612 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:00.266578 2569 generic.go:356] "Generic (PLEG): container finished" podID="6172bc03-a7dd-4306-8d35-aa844d32ed66" containerID="bdfb40fe13193b7b1dd237c027b6224c2f9fc46642fb7efcfa3c40f34b09d020" exitCode=0 Jul 24 15:51:00.266785 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:00.266660 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng" event={"ID":"6172bc03-a7dd-4306-8d35-aa844d32ed66","Type":"ContainerDied","Data":"bdfb40fe13193b7b1dd237c027b6224c2f9fc46642fb7efcfa3c40f34b09d020"} Jul 24 15:51:00.266785 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:00.266709 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng" event={"ID":"6172bc03-a7dd-4306-8d35-aa844d32ed66","Type":"ContainerStarted","Data":"01407d33ed7dece9f1e5fca58db2137df1918fa5de268189132ab2692674fca4"} Jul 24 15:51:03.282365 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:03.282325 2569 generic.go:356] "Generic (PLEG): container finished" podID="6172bc03-a7dd-4306-8d35-aa844d32ed66" containerID="3dd8fa190ff5875aab3a246a57d109c2910781ca3c1cadc056b890af80c9ba66" exitCode=0 Jul 24 15:51:03.282807 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:03.282405 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng" event={"ID":"6172bc03-a7dd-4306-8d35-aa844d32ed66","Type":"ContainerDied","Data":"3dd8fa190ff5875aab3a246a57d109c2910781ca3c1cadc056b890af80c9ba66"} Jul 24 15:51:04.289529 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:04.289499 2569 generic.go:356] "Generic (PLEG): container finished" podID="6172bc03-a7dd-4306-8d35-aa844d32ed66" containerID="9d706f69212796971d743264915f0bdcf58e9e0119d6d052921ad1e102281280" exitCode=0 Jul 24 15:51:04.289912 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:04.289547 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng" event={"ID":"6172bc03-a7dd-4306-8d35-aa844d32ed66","Type":"ContainerDied","Data":"9d706f69212796971d743264915f0bdcf58e9e0119d6d052921ad1e102281280"} Jul 24 15:51:05.423312 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:05.423284 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng" Jul 24 15:51:05.525308 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:05.525261 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckqjn\" (UniqueName: \"kubernetes.io/projected/6172bc03-a7dd-4306-8d35-aa844d32ed66-kube-api-access-ckqjn\") pod \"6172bc03-a7dd-4306-8d35-aa844d32ed66\" (UID: \"6172bc03-a7dd-4306-8d35-aa844d32ed66\") " Jul 24 15:51:05.525487 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:05.525341 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6172bc03-a7dd-4306-8d35-aa844d32ed66-bundle\") pod \"6172bc03-a7dd-4306-8d35-aa844d32ed66\" (UID: \"6172bc03-a7dd-4306-8d35-aa844d32ed66\") " Jul 24 15:51:05.525487 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:05.525365 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6172bc03-a7dd-4306-8d35-aa844d32ed66-util\") pod \"6172bc03-a7dd-4306-8d35-aa844d32ed66\" (UID: \"6172bc03-a7dd-4306-8d35-aa844d32ed66\") " Jul 24 15:51:05.525734 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:05.525707 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6172bc03-a7dd-4306-8d35-aa844d32ed66-bundle" (OuterVolumeSpecName: "bundle") pod "6172bc03-a7dd-4306-8d35-aa844d32ed66" (UID: "6172bc03-a7dd-4306-8d35-aa844d32ed66"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:51:05.527404 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:05.527373 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6172bc03-a7dd-4306-8d35-aa844d32ed66-kube-api-access-ckqjn" (OuterVolumeSpecName: "kube-api-access-ckqjn") pod "6172bc03-a7dd-4306-8d35-aa844d32ed66" (UID: "6172bc03-a7dd-4306-8d35-aa844d32ed66"). InnerVolumeSpecName "kube-api-access-ckqjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:51:05.529802 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:05.529779 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6172bc03-a7dd-4306-8d35-aa844d32ed66-util" (OuterVolumeSpecName: "util") pod "6172bc03-a7dd-4306-8d35-aa844d32ed66" (UID: "6172bc03-a7dd-4306-8d35-aa844d32ed66"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:51:05.626008 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:05.625926 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckqjn\" (UniqueName: \"kubernetes.io/projected/6172bc03-a7dd-4306-8d35-aa844d32ed66-kube-api-access-ckqjn\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:51:05.626008 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:05.625951 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6172bc03-a7dd-4306-8d35-aa844d32ed66-bundle\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:51:05.626008 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:05.625963 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6172bc03-a7dd-4306-8d35-aa844d32ed66-util\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:51:06.300673 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:06.300646 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng" Jul 24 15:51:06.300843 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:06.300637 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fbs9ng" event={"ID":"6172bc03-a7dd-4306-8d35-aa844d32ed66","Type":"ContainerDied","Data":"01407d33ed7dece9f1e5fca58db2137df1918fa5de268189132ab2692674fca4"} Jul 24 15:51:06.300843 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:06.300763 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01407d33ed7dece9f1e5fca58db2137df1918fa5de268189132ab2692674fca4" Jul 24 15:51:11.010982 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.010947 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mvmkt"] Jul 24 15:51:11.011523 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.011503 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6172bc03-a7dd-4306-8d35-aa844d32ed66" containerName="extract" Jul 24 15:51:11.011596 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.011526 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6172bc03-a7dd-4306-8d35-aa844d32ed66" containerName="extract" Jul 24 15:51:11.011596 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.011561 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6172bc03-a7dd-4306-8d35-aa844d32ed66" containerName="pull" Jul 24 15:51:11.011596 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.011570 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6172bc03-a7dd-4306-8d35-aa844d32ed66" containerName="pull" Jul 24 15:51:11.011596 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.011585 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6172bc03-a7dd-4306-8d35-aa844d32ed66" containerName="util" Jul 24 15:51:11.011596 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.011593 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6172bc03-a7dd-4306-8d35-aa844d32ed66" containerName="util" Jul 24 15:51:11.011826 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.011675 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6172bc03-a7dd-4306-8d35-aa844d32ed66" containerName="extract" Jul 24 15:51:11.013848 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.013827 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mvmkt" Jul 24 15:51:11.016350 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.016330 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 24 15:51:11.017366 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.017351 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-zbw5q\"" Jul 24 15:51:11.017428 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.017393 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 24 15:51:11.024449 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.024426 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mvmkt"] Jul 24 15:51:11.173302 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.173269 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/425eb251-1617-4c16-94b0-266b80f13bea-bound-sa-token\") pod \"cert-manager-55b4d954d5-mvmkt\" (UID: \"425eb251-1617-4c16-94b0-266b80f13bea\") " pod="cert-manager/cert-manager-55b4d954d5-mvmkt" Jul 24 15:51:11.173444 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.173352 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cls6v\" (UniqueName: \"kubernetes.io/projected/425eb251-1617-4c16-94b0-266b80f13bea-kube-api-access-cls6v\") pod \"cert-manager-55b4d954d5-mvmkt\" (UID: \"425eb251-1617-4c16-94b0-266b80f13bea\") " pod="cert-manager/cert-manager-55b4d954d5-mvmkt" Jul 24 15:51:11.274652 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.274573 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/425eb251-1617-4c16-94b0-266b80f13bea-bound-sa-token\") pod \"cert-manager-55b4d954d5-mvmkt\" (UID: \"425eb251-1617-4c16-94b0-266b80f13bea\") " pod="cert-manager/cert-manager-55b4d954d5-mvmkt" Jul 24 15:51:11.274652 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.274642 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cls6v\" (UniqueName: \"kubernetes.io/projected/425eb251-1617-4c16-94b0-266b80f13bea-kube-api-access-cls6v\") pod \"cert-manager-55b4d954d5-mvmkt\" (UID: \"425eb251-1617-4c16-94b0-266b80f13bea\") " pod="cert-manager/cert-manager-55b4d954d5-mvmkt" Jul 24 15:51:11.282893 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.282869 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/425eb251-1617-4c16-94b0-266b80f13bea-bound-sa-token\") pod \"cert-manager-55b4d954d5-mvmkt\" (UID: \"425eb251-1617-4c16-94b0-266b80f13bea\") " pod="cert-manager/cert-manager-55b4d954d5-mvmkt" Jul 24 15:51:11.283112 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.283093 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cls6v\" (UniqueName: \"kubernetes.io/projected/425eb251-1617-4c16-94b0-266b80f13bea-kube-api-access-cls6v\") pod \"cert-manager-55b4d954d5-mvmkt\" (UID: \"425eb251-1617-4c16-94b0-266b80f13bea\") " pod="cert-manager/cert-manager-55b4d954d5-mvmkt" Jul 24 15:51:11.338373 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.338341 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mvmkt" Jul 24 15:51:11.467656 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:11.467630 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mvmkt"] Jul 24 15:51:11.469733 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:51:11.469706 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod425eb251_1617_4c16_94b0_266b80f13bea.slice/crio-c53f6e44dd7d696dcf116620808ef507c2c33ad27af83b860e853bca5b1c08d5 WatchSource:0}: Error finding container c53f6e44dd7d696dcf116620808ef507c2c33ad27af83b860e853bca5b1c08d5: Status 404 returned error can't find the container with id c53f6e44dd7d696dcf116620808ef507c2c33ad27af83b860e853bca5b1c08d5 Jul 24 15:51:12.331649 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:12.331611 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mvmkt" event={"ID":"425eb251-1617-4c16-94b0-266b80f13bea","Type":"ContainerStarted","Data":"c53f6e44dd7d696dcf116620808ef507c2c33ad27af83b860e853bca5b1c08d5"} Jul 24 15:51:12.387989 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:12.387948 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s9ndj"] Jul 24 15:51:12.390198 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:12.390176 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s9ndj" Jul 24 15:51:12.393077 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:12.393057 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 24 15:51:12.394036 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:12.394014 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6jdff\"" Jul 24 15:51:12.394036 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:12.394025 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:51:12.399070 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:12.399046 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s9ndj"] Jul 24 15:51:12.487233 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:12.487179 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv66q\" (UniqueName: \"kubernetes.io/projected/2176d9a3-1ea1-4527-916b-0231cf449d39-kube-api-access-mv66q\") pod \"openshift-lws-operator-fd8ccff4c-s9ndj\" (UID: \"2176d9a3-1ea1-4527-916b-0231cf449d39\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s9ndj" Jul 24 15:51:12.487420 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:12.487250 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2176d9a3-1ea1-4527-916b-0231cf449d39-tmp\") pod \"openshift-lws-operator-fd8ccff4c-s9ndj\" (UID: \"2176d9a3-1ea1-4527-916b-0231cf449d39\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s9ndj" Jul 24 15:51:12.588654 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:12.588559 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv66q\" (UniqueName: \"kubernetes.io/projected/2176d9a3-1ea1-4527-916b-0231cf449d39-kube-api-access-mv66q\") pod \"openshift-lws-operator-fd8ccff4c-s9ndj\" (UID: \"2176d9a3-1ea1-4527-916b-0231cf449d39\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s9ndj" Jul 24 15:51:12.588654 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:12.588618 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2176d9a3-1ea1-4527-916b-0231cf449d39-tmp\") pod \"openshift-lws-operator-fd8ccff4c-s9ndj\" (UID: \"2176d9a3-1ea1-4527-916b-0231cf449d39\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s9ndj" Jul 24 15:51:12.589006 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:12.588988 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2176d9a3-1ea1-4527-916b-0231cf449d39-tmp\") pod \"openshift-lws-operator-fd8ccff4c-s9ndj\" (UID: \"2176d9a3-1ea1-4527-916b-0231cf449d39\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s9ndj" Jul 24 15:51:12.597954 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:12.597930 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv66q\" (UniqueName: \"kubernetes.io/projected/2176d9a3-1ea1-4527-916b-0231cf449d39-kube-api-access-mv66q\") pod \"openshift-lws-operator-fd8ccff4c-s9ndj\" (UID: \"2176d9a3-1ea1-4527-916b-0231cf449d39\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s9ndj" Jul 24 15:51:12.703544 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:12.703511 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s9ndj" Jul 24 15:51:12.858019 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:12.857987 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s9ndj"] Jul 24 15:51:12.860445 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:51:12.860410 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2176d9a3_1ea1_4527_916b_0231cf449d39.slice/crio-cdbce7ed443bcb179268e38e5ecfbf0b2afc2a70accb243e0a9db16b86b774e9 WatchSource:0}: Error finding container cdbce7ed443bcb179268e38e5ecfbf0b2afc2a70accb243e0a9db16b86b774e9: Status 404 returned error can't find the container with id cdbce7ed443bcb179268e38e5ecfbf0b2afc2a70accb243e0a9db16b86b774e9 Jul 24 15:51:13.339840 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:13.339802 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s9ndj" event={"ID":"2176d9a3-1ea1-4527-916b-0231cf449d39","Type":"ContainerStarted","Data":"cdbce7ed443bcb179268e38e5ecfbf0b2afc2a70accb243e0a9db16b86b774e9"} Jul 24 15:51:14.348048 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:14.348007 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mvmkt" event={"ID":"425eb251-1617-4c16-94b0-266b80f13bea","Type":"ContainerStarted","Data":"8d382eb17fc39ee6f7b0236a7e9bef1eaa0fc8b431070f06d308ca1032984042"} Jul 24 15:51:14.366664 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:14.366615 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-mvmkt" podStartSLOduration=2.002221621 podStartE2EDuration="4.366597666s" totalImagesPullingTime="2.364376045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:51:14.364175164 +0000 UTC m=+349.282145465" watchObservedRunningTime="2026-07-24 15:51:14.366597666 +0000 UTC m=+349.284567966" Jul 24 15:51:16.359565 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:16.359526 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s9ndj" event={"ID":"2176d9a3-1ea1-4527-916b-0231cf449d39","Type":"ContainerStarted","Data":"df2575896d634c24454395a7d007df138c3cc6c5fd0840405955386df0dcc8da"} Jul 24 15:51:16.382423 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:16.382372 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-s9ndj" podStartSLOduration=1.859958244 podStartE2EDuration="4.382356559s" totalImagesPullingTime="2.522398315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:51:16.380992227 +0000 UTC m=+351.298962527" watchObservedRunningTime="2026-07-24 15:51:16.382356559 +0000 UTC m=+351.300326858" Jul 24 15:51:20.040753 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:20.040663 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k"] Jul 24 15:51:20.044362 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:20.044332 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k" Jul 24 15:51:20.047268 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:20.047246 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 15:51:20.048293 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:20.048272 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 15:51:20.048410 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:20.048284 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jnzpn\"" Jul 24 15:51:20.055014 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:20.054989 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k"] Jul 24 15:51:20.059835 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:20.059813 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9a72ae8-0225-4a0b-9226-880cdfdb384b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k\" (UID: \"f9a72ae8-0225-4a0b-9226-880cdfdb384b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k" Jul 24 15:51:20.059936 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:20.059846 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9a72ae8-0225-4a0b-9226-880cdfdb384b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k\" (UID: \"f9a72ae8-0225-4a0b-9226-880cdfdb384b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k" Jul 24 15:51:20.059936 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:20.059874 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qckkw\" (UniqueName: \"kubernetes.io/projected/f9a72ae8-0225-4a0b-9226-880cdfdb384b-kube-api-access-qckkw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k\" (UID: \"f9a72ae8-0225-4a0b-9226-880cdfdb384b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k" Jul 24 15:51:20.160672 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:20.160642 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9a72ae8-0225-4a0b-9226-880cdfdb384b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k\" (UID: \"f9a72ae8-0225-4a0b-9226-880cdfdb384b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k" Jul 24 15:51:20.160842 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:20.160680 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9a72ae8-0225-4a0b-9226-880cdfdb384b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k\" (UID: \"f9a72ae8-0225-4a0b-9226-880cdfdb384b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k" Jul 24 15:51:20.160842 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:20.160713 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qckkw\" (UniqueName: \"kubernetes.io/projected/f9a72ae8-0225-4a0b-9226-880cdfdb384b-kube-api-access-qckkw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k\" (UID: \"f9a72ae8-0225-4a0b-9226-880cdfdb384b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k" Jul 24 15:51:20.161132 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:20.161107 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9a72ae8-0225-4a0b-9226-880cdfdb384b-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k\" (UID: \"f9a72ae8-0225-4a0b-9226-880cdfdb384b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k" Jul 24 15:51:20.161171 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:20.161109 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9a72ae8-0225-4a0b-9226-880cdfdb384b-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k\" (UID: \"f9a72ae8-0225-4a0b-9226-880cdfdb384b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k" Jul 24 15:51:20.169302 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:20.169269 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qckkw\" (UniqueName: \"kubernetes.io/projected/f9a72ae8-0225-4a0b-9226-880cdfdb384b-kube-api-access-qckkw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k\" (UID: \"f9a72ae8-0225-4a0b-9226-880cdfdb384b\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k" Jul 24 15:51:20.354437 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:20.354342 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k" Jul 24 15:51:20.484277 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:20.484253 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k"] Jul 24 15:51:20.487083 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:51:20.487054 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a72ae8_0225_4a0b_9226_880cdfdb384b.slice/crio-af1ada648911317705299b3b6613e25a2dc9978b0524b0e1c6f2be6eedcf4a16 WatchSource:0}: Error finding container af1ada648911317705299b3b6613e25a2dc9978b0524b0e1c6f2be6eedcf4a16: Status 404 returned error can't find the container with id af1ada648911317705299b3b6613e25a2dc9978b0524b0e1c6f2be6eedcf4a16 Jul 24 15:51:21.385542 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:21.385510 2569 generic.go:356] "Generic (PLEG): container finished" podID="f9a72ae8-0225-4a0b-9226-880cdfdb384b" containerID="f4a4561b6a30b52d47e12609ee299894f24d515b37b5e64d768a691f1cede7a1" exitCode=0 Jul 24 15:51:21.385922 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:21.385551 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k" event={"ID":"f9a72ae8-0225-4a0b-9226-880cdfdb384b","Type":"ContainerDied","Data":"f4a4561b6a30b52d47e12609ee299894f24d515b37b5e64d768a691f1cede7a1"} Jul 24 15:51:21.385922 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:21.385586 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k" event={"ID":"f9a72ae8-0225-4a0b-9226-880cdfdb384b","Type":"ContainerStarted","Data":"af1ada648911317705299b3b6613e25a2dc9978b0524b0e1c6f2be6eedcf4a16"} Jul 24 15:51:22.392736 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:22.392688 2569 generic.go:356] "Generic (PLEG): container finished" podID="f9a72ae8-0225-4a0b-9226-880cdfdb384b" containerID="5da6a835ab588c5c650175f0d0f12e6be640a7d87b2ee002aec6575225393c30" exitCode=0 Jul 24 15:51:22.393111 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:22.392752 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k" event={"ID":"f9a72ae8-0225-4a0b-9226-880cdfdb384b","Type":"ContainerDied","Data":"5da6a835ab588c5c650175f0d0f12e6be640a7d87b2ee002aec6575225393c30"} Jul 24 15:51:23.401897 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:23.401862 2569 generic.go:356] "Generic (PLEG): container finished" podID="f9a72ae8-0225-4a0b-9226-880cdfdb384b" containerID="599481ad86c25bda038399ec2573e39c826e07c51494d45c1fef7d4f3c2ff69c" exitCode=0 Jul 24 15:51:23.402306 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:23.401946 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k" event={"ID":"f9a72ae8-0225-4a0b-9226-880cdfdb384b","Type":"ContainerDied","Data":"599481ad86c25bda038399ec2573e39c826e07c51494d45c1fef7d4f3c2ff69c"} Jul 24 15:51:24.536366 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:24.536341 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k" Jul 24 15:51:24.596127 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:24.596096 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qckkw\" (UniqueName: \"kubernetes.io/projected/f9a72ae8-0225-4a0b-9226-880cdfdb384b-kube-api-access-qckkw\") pod \"f9a72ae8-0225-4a0b-9226-880cdfdb384b\" (UID: \"f9a72ae8-0225-4a0b-9226-880cdfdb384b\") " Jul 24 15:51:24.596314 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:24.596139 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9a72ae8-0225-4a0b-9226-880cdfdb384b-bundle\") pod \"f9a72ae8-0225-4a0b-9226-880cdfdb384b\" (UID: \"f9a72ae8-0225-4a0b-9226-880cdfdb384b\") " Jul 24 15:51:24.596365 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:24.596308 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9a72ae8-0225-4a0b-9226-880cdfdb384b-util\") pod \"f9a72ae8-0225-4a0b-9226-880cdfdb384b\" (UID: \"f9a72ae8-0225-4a0b-9226-880cdfdb384b\") " Jul 24 15:51:24.596864 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:24.596841 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a72ae8-0225-4a0b-9226-880cdfdb384b-bundle" (OuterVolumeSpecName: "bundle") pod "f9a72ae8-0225-4a0b-9226-880cdfdb384b" (UID: "f9a72ae8-0225-4a0b-9226-880cdfdb384b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:51:24.598206 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:24.598187 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a72ae8-0225-4a0b-9226-880cdfdb384b-kube-api-access-qckkw" (OuterVolumeSpecName: "kube-api-access-qckkw") pod "f9a72ae8-0225-4a0b-9226-880cdfdb384b" (UID: "f9a72ae8-0225-4a0b-9226-880cdfdb384b"). InnerVolumeSpecName "kube-api-access-qckkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:51:24.604368 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:24.604343 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a72ae8-0225-4a0b-9226-880cdfdb384b-util" (OuterVolumeSpecName: "util") pod "f9a72ae8-0225-4a0b-9226-880cdfdb384b" (UID: "f9a72ae8-0225-4a0b-9226-880cdfdb384b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:51:24.697833 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:24.697743 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f9a72ae8-0225-4a0b-9226-880cdfdb384b-util\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:51:24.697833 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:24.697776 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qckkw\" (UniqueName: \"kubernetes.io/projected/f9a72ae8-0225-4a0b-9226-880cdfdb384b-kube-api-access-qckkw\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:51:24.697833 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:24.697786 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f9a72ae8-0225-4a0b-9226-880cdfdb384b-bundle\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:51:25.414235 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:25.414191 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k" event={"ID":"f9a72ae8-0225-4a0b-9226-880cdfdb384b","Type":"ContainerDied","Data":"af1ada648911317705299b3b6613e25a2dc9978b0524b0e1c6f2be6eedcf4a16"} Jul 24 15:51:25.414235 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:25.414228 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwlm7k" Jul 24 15:51:25.414441 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:25.414251 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af1ada648911317705299b3b6613e25a2dc9978b0524b0e1c6f2be6eedcf4a16" Jul 24 15:51:32.620440 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.620402 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs"] Jul 24 15:51:32.620875 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.620772 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9a72ae8-0225-4a0b-9226-880cdfdb384b" containerName="pull" Jul 24 15:51:32.620875 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.620784 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9a72ae8-0225-4a0b-9226-880cdfdb384b" containerName="pull" Jul 24 15:51:32.620875 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.620792 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9a72ae8-0225-4a0b-9226-880cdfdb384b" containerName="extract" Jul 24 15:51:32.620875 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.620798 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9a72ae8-0225-4a0b-9226-880cdfdb384b" containerName="extract" Jul 24 15:51:32.620875 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.620813 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9a72ae8-0225-4a0b-9226-880cdfdb384b" containerName="util" Jul 24 15:51:32.620875 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.620818 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9a72ae8-0225-4a0b-9226-880cdfdb384b" containerName="util" Jul 24 15:51:32.621055 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.620884 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9a72ae8-0225-4a0b-9226-880cdfdb384b" containerName="extract" Jul 24 15:51:32.628086 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.628068 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" Jul 24 15:51:32.630976 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.630955 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 15:51:32.631099 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.631012 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 15:51:32.631937 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.631920 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8dkpp\"" Jul 24 15:51:32.632024 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.631958 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 15:51:32.632084 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.632028 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 15:51:32.641575 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.641553 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs"] Jul 24 15:51:32.667902 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.667869 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/68181f3b-f19f-459f-b325-4bedfcd80b77-manifests\") pod \"opendatahub-operator-controller-manager-5d995667b8-fbtbs\" (UID: \"68181f3b-f19f-459f-b325-4bedfcd80b77\") " pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" Jul 24 15:51:32.668080 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.667911 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68181f3b-f19f-459f-b325-4bedfcd80b77-tmp\") pod \"opendatahub-operator-controller-manager-5d995667b8-fbtbs\" (UID: \"68181f3b-f19f-459f-b325-4bedfcd80b77\") " pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" Jul 24 15:51:32.668080 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.668002 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skvtj\" (UniqueName: \"kubernetes.io/projected/68181f3b-f19f-459f-b325-4bedfcd80b77-kube-api-access-skvtj\") pod \"opendatahub-operator-controller-manager-5d995667b8-fbtbs\" (UID: \"68181f3b-f19f-459f-b325-4bedfcd80b77\") " pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" Jul 24 15:51:32.668080 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.668056 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/68181f3b-f19f-459f-b325-4bedfcd80b77-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d995667b8-fbtbs\" (UID: \"68181f3b-f19f-459f-b325-4bedfcd80b77\") " pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" Jul 24 15:51:32.668241 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.668083 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68181f3b-f19f-459f-b325-4bedfcd80b77-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d995667b8-fbtbs\" (UID: \"68181f3b-f19f-459f-b325-4bedfcd80b77\") " pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" Jul 24 15:51:32.769298 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.769259 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skvtj\" (UniqueName: \"kubernetes.io/projected/68181f3b-f19f-459f-b325-4bedfcd80b77-kube-api-access-skvtj\") pod \"opendatahub-operator-controller-manager-5d995667b8-fbtbs\" (UID: \"68181f3b-f19f-459f-b325-4bedfcd80b77\") " pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" Jul 24 15:51:32.769469 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.769318 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/68181f3b-f19f-459f-b325-4bedfcd80b77-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d995667b8-fbtbs\" (UID: \"68181f3b-f19f-459f-b325-4bedfcd80b77\") " pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" Jul 24 15:51:32.769469 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.769347 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68181f3b-f19f-459f-b325-4bedfcd80b77-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d995667b8-fbtbs\" (UID: \"68181f3b-f19f-459f-b325-4bedfcd80b77\") " pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" Jul 24 15:51:32.769469 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.769368 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/68181f3b-f19f-459f-b325-4bedfcd80b77-manifests\") pod \"opendatahub-operator-controller-manager-5d995667b8-fbtbs\" (UID: \"68181f3b-f19f-459f-b325-4bedfcd80b77\") " pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" Jul 24 15:51:32.769469 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.769394 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68181f3b-f19f-459f-b325-4bedfcd80b77-tmp\") pod \"opendatahub-operator-controller-manager-5d995667b8-fbtbs\" (UID: \"68181f3b-f19f-459f-b325-4bedfcd80b77\") " pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" Jul 24 15:51:32.769798 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.769778 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/68181f3b-f19f-459f-b325-4bedfcd80b77-manifests\") pod \"opendatahub-operator-controller-manager-5d995667b8-fbtbs\" (UID: \"68181f3b-f19f-459f-b325-4bedfcd80b77\") " pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" Jul 24 15:51:32.771842 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.771816 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68181f3b-f19f-459f-b325-4bedfcd80b77-tmp\") pod \"opendatahub-operator-controller-manager-5d995667b8-fbtbs\" (UID: \"68181f3b-f19f-459f-b325-4bedfcd80b77\") " pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" Jul 24 15:51:32.771983 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.771961 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/68181f3b-f19f-459f-b325-4bedfcd80b77-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d995667b8-fbtbs\" (UID: \"68181f3b-f19f-459f-b325-4bedfcd80b77\") " pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" Jul 24 15:51:32.772050 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.771987 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/68181f3b-f19f-459f-b325-4bedfcd80b77-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d995667b8-fbtbs\" (UID: \"68181f3b-f19f-459f-b325-4bedfcd80b77\") " pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" Jul 24 15:51:32.777418 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.777392 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skvtj\" (UniqueName: \"kubernetes.io/projected/68181f3b-f19f-459f-b325-4bedfcd80b77-kube-api-access-skvtj\") pod \"opendatahub-operator-controller-manager-5d995667b8-fbtbs\" (UID: \"68181f3b-f19f-459f-b325-4bedfcd80b77\") " pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" Jul 24 15:51:32.938360 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:32.938329 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" Jul 24 15:51:33.080528 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:33.080502 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs"] Jul 24 15:51:33.081939 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:51:33.081917 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68181f3b_f19f_459f_b325_4bedfcd80b77.slice/crio-a779bc5037ed90db9c30058276b3ad495990b52df6c5d4b42b410a41373b9a6e WatchSource:0}: Error finding container a779bc5037ed90db9c30058276b3ad495990b52df6c5d4b42b410a41373b9a6e: Status 404 returned error can't find the container with id a779bc5037ed90db9c30058276b3ad495990b52df6c5d4b42b410a41373b9a6e Jul 24 15:51:33.452511 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:33.452473 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" event={"ID":"68181f3b-f19f-459f-b325-4bedfcd80b77","Type":"ContainerStarted","Data":"a779bc5037ed90db9c30058276b3ad495990b52df6c5d4b42b410a41373b9a6e"} Jul 24 15:51:36.470168 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:36.470135 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" event={"ID":"68181f3b-f19f-459f-b325-4bedfcd80b77","Type":"ContainerStarted","Data":"496229a07244ba4d382a307adadddf9c55f1ba87ba9f29cb147f5f3fcc71c737"} Jul 24 15:51:37.477006 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:37.476969 2569 generic.go:356] "Generic (PLEG): container finished" podID="68181f3b-f19f-459f-b325-4bedfcd80b77" containerID="496229a07244ba4d382a307adadddf9c55f1ba87ba9f29cb147f5f3fcc71c737" exitCode=0 Jul 24 15:51:37.477507 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:37.477048 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" event={"ID":"68181f3b-f19f-459f-b325-4bedfcd80b77","Type":"ContainerDied","Data":"496229a07244ba4d382a307adadddf9c55f1ba87ba9f29cb147f5f3fcc71c737"} Jul 24 15:51:38.484359 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:38.484325 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" event={"ID":"68181f3b-f19f-459f-b325-4bedfcd80b77","Type":"ContainerStarted","Data":"396d8b66959c9f3d160cbe5f3993e74b38a5ddd78bb80425ec8806b0aeb2ecea"} Jul 24 15:51:38.484732 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:38.484466 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" Jul 24 15:51:38.507661 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:38.507616 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" podStartSLOduration=2.116535592 podStartE2EDuration="6.507596473s" totalImagesPullingTime="3.344391216s" totalInitContainerRuntime="1.046669665s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:51:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:51:38.505193401 +0000 UTC m=+373.423163701" watchObservedRunningTime="2026-07-24 15:51:38.507596473 +0000 UTC m=+373.425566774" Jul 24 15:51:49.491892 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:49.491861 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d995667b8-fbtbs" Jul 24 15:51:52.019741 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.019709 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-767449f4cf-hgvr5"] Jul 24 15:51:52.023126 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.023105 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-767449f4cf-hgvr5" Jul 24 15:51:52.026859 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.026837 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 24 15:51:52.026859 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.026846 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 24 15:51:52.027045 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.026837 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8vnkm\"" Jul 24 15:51:52.037881 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.037855 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-767449f4cf-hgvr5"] Jul 24 15:51:52.144324 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.144292 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/319718c8-0b89-4ecf-ae6b-03e13f89c9fb-tmp\") pod \"kube-auth-proxy-767449f4cf-hgvr5\" (UID: \"319718c8-0b89-4ecf-ae6b-03e13f89c9fb\") " pod="openshift-ingress/kube-auth-proxy-767449f4cf-hgvr5" Jul 24 15:51:52.144516 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.144330 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz4rk\" (UniqueName: \"kubernetes.io/projected/319718c8-0b89-4ecf-ae6b-03e13f89c9fb-kube-api-access-fz4rk\") pod \"kube-auth-proxy-767449f4cf-hgvr5\" (UID: \"319718c8-0b89-4ecf-ae6b-03e13f89c9fb\") " pod="openshift-ingress/kube-auth-proxy-767449f4cf-hgvr5" Jul 24 15:51:52.144516 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.144364 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/319718c8-0b89-4ecf-ae6b-03e13f89c9fb-tls-certs\") pod \"kube-auth-proxy-767449f4cf-hgvr5\" (UID: \"319718c8-0b89-4ecf-ae6b-03e13f89c9fb\") " pod="openshift-ingress/kube-auth-proxy-767449f4cf-hgvr5" Jul 24 15:51:52.244890 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.244857 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/319718c8-0b89-4ecf-ae6b-03e13f89c9fb-tmp\") pod \"kube-auth-proxy-767449f4cf-hgvr5\" (UID: \"319718c8-0b89-4ecf-ae6b-03e13f89c9fb\") " pod="openshift-ingress/kube-auth-proxy-767449f4cf-hgvr5" Jul 24 15:51:52.244890 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.244892 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz4rk\" (UniqueName: \"kubernetes.io/projected/319718c8-0b89-4ecf-ae6b-03e13f89c9fb-kube-api-access-fz4rk\") pod \"kube-auth-proxy-767449f4cf-hgvr5\" (UID: \"319718c8-0b89-4ecf-ae6b-03e13f89c9fb\") " pod="openshift-ingress/kube-auth-proxy-767449f4cf-hgvr5" Jul 24 15:51:52.245131 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.244920 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/319718c8-0b89-4ecf-ae6b-03e13f89c9fb-tls-certs\") pod \"kube-auth-proxy-767449f4cf-hgvr5\" (UID: \"319718c8-0b89-4ecf-ae6b-03e13f89c9fb\") " pod="openshift-ingress/kube-auth-proxy-767449f4cf-hgvr5" Jul 24 15:51:52.247145 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.247114 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/319718c8-0b89-4ecf-ae6b-03e13f89c9fb-tmp\") pod \"kube-auth-proxy-767449f4cf-hgvr5\" (UID: \"319718c8-0b89-4ecf-ae6b-03e13f89c9fb\") " pod="openshift-ingress/kube-auth-proxy-767449f4cf-hgvr5" Jul 24 15:51:52.247431 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.247413 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/319718c8-0b89-4ecf-ae6b-03e13f89c9fb-tls-certs\") pod \"kube-auth-proxy-767449f4cf-hgvr5\" (UID: \"319718c8-0b89-4ecf-ae6b-03e13f89c9fb\") " pod="openshift-ingress/kube-auth-proxy-767449f4cf-hgvr5" Jul 24 15:51:52.257139 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.257116 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz4rk\" (UniqueName: \"kubernetes.io/projected/319718c8-0b89-4ecf-ae6b-03e13f89c9fb-kube-api-access-fz4rk\") pod \"kube-auth-proxy-767449f4cf-hgvr5\" (UID: \"319718c8-0b89-4ecf-ae6b-03e13f89c9fb\") " pod="openshift-ingress/kube-auth-proxy-767449f4cf-hgvr5" Jul 24 15:51:52.314617 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.314533 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kp9qs"] Jul 24 15:51:52.317160 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.317139 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-kp9qs" Jul 24 15:51:52.319879 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.319854 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 24 15:51:52.319965 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.319922 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-9fzrt\"" Jul 24 15:51:52.329917 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.329892 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kp9qs"] Jul 24 15:51:52.334799 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.334781 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-767449f4cf-hgvr5" Jul 24 15:51:52.446971 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.446936 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f-cert\") pod \"odh-model-controller-7998c67dc4-kp9qs\" (UID: \"c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-kp9qs" Jul 24 15:51:52.447184 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.447021 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzr97\" (UniqueName: \"kubernetes.io/projected/c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f-kube-api-access-xzr97\") pod \"odh-model-controller-7998c67dc4-kp9qs\" (UID: \"c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-kp9qs" Jul 24 15:51:52.482823 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.482795 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-767449f4cf-hgvr5"] Jul 24 15:51:52.485042 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:51:52.485015 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319718c8_0b89_4ecf_ae6b_03e13f89c9fb.slice/crio-aa19cdcc195f7a1dcf4ce9e9300760103c73ddb0011bab56233ee74c296d3fd7 WatchSource:0}: Error finding container aa19cdcc195f7a1dcf4ce9e9300760103c73ddb0011bab56233ee74c296d3fd7: Status 404 returned error can't find the container with id aa19cdcc195f7a1dcf4ce9e9300760103c73ddb0011bab56233ee74c296d3fd7 Jul 24 15:51:52.547908 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.547874 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f-cert\") pod \"odh-model-controller-7998c67dc4-kp9qs\" (UID: \"c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-kp9qs" Jul 24 15:51:52.548086 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.547929 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzr97\" (UniqueName: \"kubernetes.io/projected/c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f-kube-api-access-xzr97\") pod \"odh-model-controller-7998c67dc4-kp9qs\" (UID: \"c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-kp9qs" Jul 24 15:51:52.548086 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:51:52.548051 2569 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 24 15:51:52.548243 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:51:52.548129 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f-cert podName:c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f nodeName:}" failed. No retries permitted until 2026-07-24 15:51:53.048106979 +0000 UTC m=+387.966077271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f-cert") pod "odh-model-controller-7998c67dc4-kp9qs" (UID: "c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f") : secret "odh-model-controller-webhook-cert" not found Jul 24 15:51:52.555425 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.555390 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-767449f4cf-hgvr5" event={"ID":"319718c8-0b89-4ecf-ae6b-03e13f89c9fb","Type":"ContainerStarted","Data":"aa19cdcc195f7a1dcf4ce9e9300760103c73ddb0011bab56233ee74c296d3fd7"} Jul 24 15:51:52.574538 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:52.574462 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzr97\" (UniqueName: \"kubernetes.io/projected/c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f-kube-api-access-xzr97\") pod \"odh-model-controller-7998c67dc4-kp9qs\" (UID: \"c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-kp9qs" Jul 24 15:51:53.052869 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.052835 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f-cert\") pod \"odh-model-controller-7998c67dc4-kp9qs\" (UID: \"c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-kp9qs" Jul 24 15:51:53.055372 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.055344 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f-cert\") pod \"odh-model-controller-7998c67dc4-kp9qs\" (UID: \"c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-kp9qs" Jul 24 15:51:53.228616 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.228571 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-kp9qs" Jul 24 15:51:53.310684 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.310601 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4"] Jul 24 15:51:53.317918 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.317891 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.320890 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.320858 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 24 15:51:53.321046 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.320898 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 24 15:51:53.321135 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.321114 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 15:51:53.321254 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.321230 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sbx6j\"" Jul 24 15:51:53.321329 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.321117 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 24 15:51:53.330622 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.330592 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4"] Jul 24 15:51:53.438235 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.438186 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-kp9qs"] Jul 24 15:51:53.441897 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:51:53.441865 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5848b14_2d4b_4cee_b2d4_6ed2f1f66c5f.slice/crio-8ee85bf0e1ed168d5eef3568f6c11be61ae0b2bbdf329b7b912cce97163627ac WatchSource:0}: Error finding container 8ee85bf0e1ed168d5eef3568f6c11be61ae0b2bbdf329b7b912cce97163627ac: Status 404 returned error can't find the container with id 8ee85bf0e1ed168d5eef3568f6c11be61ae0b2bbdf329b7b912cce97163627ac Jul 24 15:51:53.465552 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.465519 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.465710 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.465656 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5cab2266-896c-41fb-9365-d7cf84a47c4c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.465710 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.465688 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cab2266-896c-41fb-9365-d7cf84a47c4c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.465827 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.465719 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2vzh\" (UniqueName: \"kubernetes.io/projected/5cab2266-896c-41fb-9365-d7cf84a47c4c-kube-api-access-r2vzh\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.465827 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.465785 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.465928 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.465843 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.465928 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.465901 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.563179 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.563096 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kp9qs" event={"ID":"c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f","Type":"ContainerStarted","Data":"8ee85bf0e1ed168d5eef3568f6c11be61ae0b2bbdf329b7b912cce97163627ac"} Jul 24 15:51:53.566594 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.566565 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5cab2266-896c-41fb-9365-d7cf84a47c4c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.566737 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.566599 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cab2266-896c-41fb-9365-d7cf84a47c4c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.566737 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.566620 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2vzh\" (UniqueName: \"kubernetes.io/projected/5cab2266-896c-41fb-9365-d7cf84a47c4c-kube-api-access-r2vzh\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.566737 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.566646 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.566737 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.566694 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.566737 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.566738 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.567021 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.566788 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.567726 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.567700 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.569928 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.569901 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.569928 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.569919 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cab2266-896c-41fb-9365-d7cf84a47c4c-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.570607 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.570580 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5cab2266-896c-41fb-9365-d7cf84a47c4c-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.571035 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.571017 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.576595 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.576572 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.576702 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.576681 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2vzh\" (UniqueName: \"kubernetes.io/projected/5cab2266-896c-41fb-9365-d7cf84a47c4c-kube-api-access-r2vzh\") pod \"istiod-openshift-gateway-7b7995d5cd-p42n4\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.636325 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.636297 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:53.808653 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:53.808622 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4"] Jul 24 15:51:53.809129 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:51:53.809101 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cab2266_896c_41fb_9365_d7cf84a47c4c.slice/crio-08fa7679cbc7e541d22c1a238629612a5acbdbdbde2446b03a9fe2b538ab4741 WatchSource:0}: Error finding container 08fa7679cbc7e541d22c1a238629612a5acbdbdbde2446b03a9fe2b538ab4741: Status 404 returned error can't find the container with id 08fa7679cbc7e541d22c1a238629612a5acbdbdbde2446b03a9fe2b538ab4741 Jul 24 15:51:54.573517 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:54.573448 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" event={"ID":"5cab2266-896c-41fb-9365-d7cf84a47c4c","Type":"ContainerStarted","Data":"08fa7679cbc7e541d22c1a238629612a5acbdbdbde2446b03a9fe2b538ab4741"} Jul 24 15:51:57.827845 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:57.827794 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 15:51:57.828150 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:57.827874 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 15:51:58.600824 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:58.600788 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-767449f4cf-hgvr5" event={"ID":"319718c8-0b89-4ecf-ae6b-03e13f89c9fb","Type":"ContainerStarted","Data":"bb716f59a1b6299a2ae175eda0e82c2df31a928eb96cb76aae699d1a0fc35fb9"} Jul 24 15:51:58.602514 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:58.602481 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" event={"ID":"5cab2266-896c-41fb-9365-d7cf84a47c4c","Type":"ContainerStarted","Data":"41591a3620a934295b48b697247f603dae704c3e10fbea7758255d98609d332d"} Jul 24 15:51:58.602667 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:58.602644 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:58.604600 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:58.604568 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-kp9qs" event={"ID":"c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f","Type":"ContainerStarted","Data":"574a52fceaa786b625f12fb3622681ec257dde83b2574b35abcdcad6553a5a38"} Jul 24 15:51:58.604711 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:58.604640 2569 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-p42n4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 15:51:58.604785 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:58.604713 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" podUID="5cab2266-896c-41fb-9365-d7cf84a47c4c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 15:51:58.604785 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:58.604721 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-kp9qs" Jul 24 15:51:58.618868 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:58.618808 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-767449f4cf-hgvr5" podStartSLOduration=2.339204702 podStartE2EDuration="7.618788881s" totalImagesPullingTime="5.279584179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:51:58.617716285 +0000 UTC m=+393.535686593" watchObservedRunningTime="2026-07-24 15:51:58.618788881 +0000 UTC m=+393.536759185" Jul 24 15:51:58.643796 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:58.643726 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" podStartSLOduration=1.627837166 podStartE2EDuration="5.64370537s" totalImagesPullingTime="4.015868204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:51:58.63952363 +0000 UTC m=+393.557493930" watchObservedRunningTime="2026-07-24 15:51:58.64370537 +0000 UTC m=+393.561675672" Jul 24 15:51:59.610347 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:59.610317 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:51:59.632729 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:51:59.632672 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-kp9qs" podStartSLOduration=3.297942057 podStartE2EDuration="7.632651582s" totalImagesPullingTime="4.334709525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:51:58.667658872 +0000 UTC m=+393.585629173" watchObservedRunningTime="2026-07-24 15:51:59.632651582 +0000 UTC m=+394.550621884" Jul 24 15:52:03.598946 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.598914 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jcgj8"] Jul 24 15:52:03.602508 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.602489 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qqczl"] Jul 24 15:52:03.602632 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.602617 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jcgj8" Jul 24 15:52:03.605407 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.605386 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 24 15:52:03.605574 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.605552 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-56wvh\"" Jul 24 15:52:03.605937 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.605920 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qqczl" Jul 24 15:52:03.608172 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.608156 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 24 15:52:03.608835 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.608817 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-m9zm7\"" Jul 24 15:52:03.615603 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.615580 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jcgj8"] Jul 24 15:52:03.618558 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.618536 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qqczl"] Jul 24 15:52:03.671018 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.670985 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4q8s\" (UniqueName: \"kubernetes.io/projected/4a0df429-8f7b-45ea-b167-96c39c5db85d-kube-api-access-v4q8s\") pod \"kserve-controller-manager-d6f4b65b8-jcgj8\" (UID: \"4a0df429-8f7b-45ea-b167-96c39c5db85d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jcgj8" Jul 24 15:52:03.671181 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.671041 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0134774e-c656-4108-99a4-57714cc9e9f3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qqczl\" (UID: \"0134774e-c656-4108-99a4-57714cc9e9f3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qqczl" Jul 24 15:52:03.671181 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.671068 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6b95\" (UniqueName: \"kubernetes.io/projected/0134774e-c656-4108-99a4-57714cc9e9f3-kube-api-access-x6b95\") pod \"llmisvc-controller-manager-777c6bd7d5-qqczl\" (UID: \"0134774e-c656-4108-99a4-57714cc9e9f3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qqczl" Jul 24 15:52:03.671181 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.671090 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a0df429-8f7b-45ea-b167-96c39c5db85d-cert\") pod \"kserve-controller-manager-d6f4b65b8-jcgj8\" (UID: \"4a0df429-8f7b-45ea-b167-96c39c5db85d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jcgj8" Jul 24 15:52:03.772399 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.772362 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0134774e-c656-4108-99a4-57714cc9e9f3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qqczl\" (UID: \"0134774e-c656-4108-99a4-57714cc9e9f3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qqczl" Jul 24 15:52:03.772399 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.772399 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6b95\" (UniqueName: \"kubernetes.io/projected/0134774e-c656-4108-99a4-57714cc9e9f3-kube-api-access-x6b95\") pod \"llmisvc-controller-manager-777c6bd7d5-qqczl\" (UID: \"0134774e-c656-4108-99a4-57714cc9e9f3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qqczl" Jul 24 15:52:03.772621 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.772431 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a0df429-8f7b-45ea-b167-96c39c5db85d-cert\") pod \"kserve-controller-manager-d6f4b65b8-jcgj8\" (UID: \"4a0df429-8f7b-45ea-b167-96c39c5db85d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jcgj8" Jul 24 15:52:03.772621 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.772504 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4q8s\" (UniqueName: \"kubernetes.io/projected/4a0df429-8f7b-45ea-b167-96c39c5db85d-kube-api-access-v4q8s\") pod \"kserve-controller-manager-d6f4b65b8-jcgj8\" (UID: \"4a0df429-8f7b-45ea-b167-96c39c5db85d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jcgj8" Jul 24 15:52:03.772621 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:52:03.772514 2569 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 24 15:52:03.772621 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:52:03.772583 2569 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 24 15:52:03.772621 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:52:03.772590 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0134774e-c656-4108-99a4-57714cc9e9f3-cert podName:0134774e-c656-4108-99a4-57714cc9e9f3 nodeName:}" failed. No retries permitted until 2026-07-24 15:52:04.27256868 +0000 UTC m=+399.190538966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0134774e-c656-4108-99a4-57714cc9e9f3-cert") pod "llmisvc-controller-manager-777c6bd7d5-qqczl" (UID: "0134774e-c656-4108-99a4-57714cc9e9f3") : secret "llmisvc-webhook-server-cert" not found Jul 24 15:52:03.772785 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:52:03.772665 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a0df429-8f7b-45ea-b167-96c39c5db85d-cert podName:4a0df429-8f7b-45ea-b167-96c39c5db85d nodeName:}" failed. No retries permitted until 2026-07-24 15:52:04.272649245 +0000 UTC m=+399.190619524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4a0df429-8f7b-45ea-b167-96c39c5db85d-cert") pod "kserve-controller-manager-d6f4b65b8-jcgj8" (UID: "4a0df429-8f7b-45ea-b167-96c39c5db85d") : secret "kserve-webhook-server-cert" not found Jul 24 15:52:03.782407 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.782384 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4q8s\" (UniqueName: \"kubernetes.io/projected/4a0df429-8f7b-45ea-b167-96c39c5db85d-kube-api-access-v4q8s\") pod \"kserve-controller-manager-d6f4b65b8-jcgj8\" (UID: \"4a0df429-8f7b-45ea-b167-96c39c5db85d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jcgj8" Jul 24 15:52:03.782525 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:03.782498 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6b95\" (UniqueName: \"kubernetes.io/projected/0134774e-c656-4108-99a4-57714cc9e9f3-kube-api-access-x6b95\") pod \"llmisvc-controller-manager-777c6bd7d5-qqczl\" (UID: \"0134774e-c656-4108-99a4-57714cc9e9f3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qqczl" Jul 24 15:52:04.276786 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:04.276748 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0134774e-c656-4108-99a4-57714cc9e9f3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qqczl\" (UID: \"0134774e-c656-4108-99a4-57714cc9e9f3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qqczl" Jul 24 15:52:04.276786 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:04.276788 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a0df429-8f7b-45ea-b167-96c39c5db85d-cert\") pod \"kserve-controller-manager-d6f4b65b8-jcgj8\" (UID: \"4a0df429-8f7b-45ea-b167-96c39c5db85d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jcgj8" Jul 24 15:52:04.279247 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:04.279200 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a0df429-8f7b-45ea-b167-96c39c5db85d-cert\") pod \"kserve-controller-manager-d6f4b65b8-jcgj8\" (UID: \"4a0df429-8f7b-45ea-b167-96c39c5db85d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jcgj8" Jul 24 15:52:04.279369 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:04.279200 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0134774e-c656-4108-99a4-57714cc9e9f3-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-qqczl\" (UID: \"0134774e-c656-4108-99a4-57714cc9e9f3\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qqczl" Jul 24 15:52:04.517346 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:04.517312 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jcgj8" Jul 24 15:52:04.523538 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:04.523518 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qqczl" Jul 24 15:52:04.662134 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:04.662093 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jcgj8"] Jul 24 15:52:04.664909 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:52:04.664881 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0df429_8f7b_45ea_b167_96c39c5db85d.slice/crio-7d0d99392fc7ec85ca8dfdab7f9769968d450926f5b5912b32884d3706161db4 WatchSource:0}: Error finding container 7d0d99392fc7ec85ca8dfdab7f9769968d450926f5b5912b32884d3706161db4: Status 404 returned error can't find the container with id 7d0d99392fc7ec85ca8dfdab7f9769968d450926f5b5912b32884d3706161db4 Jul 24 15:52:04.689939 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:04.689911 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-qqczl"] Jul 24 15:52:04.691933 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:52:04.691906 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0134774e_c656_4108_99a4_57714cc9e9f3.slice/crio-6d5148091a9b73d9709f16b36bef91d3b8f9ad45c1159b836f6a23e73df881c7 WatchSource:0}: Error finding container 6d5148091a9b73d9709f16b36bef91d3b8f9ad45c1159b836f6a23e73df881c7: Status 404 returned error can't find the container with id 6d5148091a9b73d9709f16b36bef91d3b8f9ad45c1159b836f6a23e73df881c7 Jul 24 15:52:05.654446 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:05.654409 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qqczl" event={"ID":"0134774e-c656-4108-99a4-57714cc9e9f3","Type":"ContainerStarted","Data":"6d5148091a9b73d9709f16b36bef91d3b8f9ad45c1159b836f6a23e73df881c7"} Jul 24 15:52:05.655716 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:05.655664 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jcgj8" event={"ID":"4a0df429-8f7b-45ea-b167-96c39c5db85d","Type":"ContainerStarted","Data":"7d0d99392fc7ec85ca8dfdab7f9769968d450926f5b5912b32884d3706161db4"} Jul 24 15:52:07.029258 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.029188 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4"] Jul 24 15:52:07.029751 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.029465 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" podUID="5cab2266-896c-41fb-9365-d7cf84a47c4c" containerName="discovery" containerID="cri-o://41591a3620a934295b48b697247f603dae704c3e10fbea7758255d98609d332d" gracePeriod=30 Jul 24 15:52:07.670976 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.670945 2569 generic.go:356] "Generic (PLEG): container finished" podID="5cab2266-896c-41fb-9365-d7cf84a47c4c" containerID="41591a3620a934295b48b697247f603dae704c3e10fbea7758255d98609d332d" exitCode=0 Jul 24 15:52:07.671083 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.671021 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" event={"ID":"5cab2266-896c-41fb-9365-d7cf84a47c4c","Type":"ContainerDied","Data":"41591a3620a934295b48b697247f603dae704c3e10fbea7758255d98609d332d"} Jul 24 15:52:07.716384 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.716351 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:52:07.811842 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.811752 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-token\") pod \"5cab2266-896c-41fb-9365-d7cf84a47c4c\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " Jul 24 15:52:07.811842 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.811803 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-csr-ca-configmap\") pod \"5cab2266-896c-41fb-9365-d7cf84a47c4c\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " Jul 24 15:52:07.811842 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.811820 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-kubeconfig\") pod \"5cab2266-896c-41fb-9365-d7cf84a47c4c\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " Jul 24 15:52:07.812147 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.812014 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cab2266-896c-41fb-9365-d7cf84a47c4c-local-certs\") pod \"5cab2266-896c-41fb-9365-d7cf84a47c4c\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " Jul 24 15:52:07.812147 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.812087 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2vzh\" (UniqueName: \"kubernetes.io/projected/5cab2266-896c-41fb-9365-d7cf84a47c4c-kube-api-access-r2vzh\") pod \"5cab2266-896c-41fb-9365-d7cf84a47c4c\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " Jul 24 15:52:07.812298 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.812186 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5cab2266-896c-41fb-9365-d7cf84a47c4c-cacerts\") pod \"5cab2266-896c-41fb-9365-d7cf84a47c4c\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " Jul 24 15:52:07.812298 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.812245 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-csr-dns-cert\") pod \"5cab2266-896c-41fb-9365-d7cf84a47c4c\" (UID: \"5cab2266-896c-41fb-9365-d7cf84a47c4c\") " Jul 24 15:52:07.812393 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.812294 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "5cab2266-896c-41fb-9365-d7cf84a47c4c" (UID: "5cab2266-896c-41fb-9365-d7cf84a47c4c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:52:07.812592 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.812553 2569 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-csr-ca-configmap\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:52:07.814366 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.814330 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cab2266-896c-41fb-9365-d7cf84a47c4c-local-certs" (OuterVolumeSpecName: "local-certs") pod "5cab2266-896c-41fb-9365-d7cf84a47c4c" (UID: "5cab2266-896c-41fb-9365-d7cf84a47c4c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:52:07.814366 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.814356 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-token" (OuterVolumeSpecName: "istio-token") pod "5cab2266-896c-41fb-9365-d7cf84a47c4c" (UID: "5cab2266-896c-41fb-9365-d7cf84a47c4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:52:07.814521 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.814427 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "5cab2266-896c-41fb-9365-d7cf84a47c4c" (UID: "5cab2266-896c-41fb-9365-d7cf84a47c4c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:52:07.814521 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.814455 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cab2266-896c-41fb-9365-d7cf84a47c4c-kube-api-access-r2vzh" (OuterVolumeSpecName: "kube-api-access-r2vzh") pod "5cab2266-896c-41fb-9365-d7cf84a47c4c" (UID: "5cab2266-896c-41fb-9365-d7cf84a47c4c"). InnerVolumeSpecName "kube-api-access-r2vzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:52:07.814643 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.814626 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cab2266-896c-41fb-9365-d7cf84a47c4c-cacerts" (OuterVolumeSpecName: "cacerts") pod "5cab2266-896c-41fb-9365-d7cf84a47c4c" (UID: "5cab2266-896c-41fb-9365-d7cf84a47c4c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:52:07.814682 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.814660 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "5cab2266-896c-41fb-9365-d7cf84a47c4c" (UID: "5cab2266-896c-41fb-9365-d7cf84a47c4c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:52:07.913829 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.913785 2569 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cab2266-896c-41fb-9365-d7cf84a47c4c-local-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:52:07.913829 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.913833 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2vzh\" (UniqueName: \"kubernetes.io/projected/5cab2266-896c-41fb-9365-d7cf84a47c4c-kube-api-access-r2vzh\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:52:07.914093 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.913852 2569 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5cab2266-896c-41fb-9365-d7cf84a47c4c-cacerts\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:52:07.914093 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.913875 2569 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-csr-dns-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:52:07.914093 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.913890 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-token\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:52:07.914093 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:07.913903 2569 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5cab2266-896c-41fb-9365-d7cf84a47c4c-istio-kubeconfig\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:52:08.678734 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:08.678697 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jcgj8" event={"ID":"4a0df429-8f7b-45ea-b167-96c39c5db85d","Type":"ContainerStarted","Data":"19b29a02aa6f9ae3fbaed0e7daa27a823bf2cf3c71924b25893a55ac8bc2c7f3"} Jul 24 15:52:08.679199 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:08.678900 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jcgj8" Jul 24 15:52:08.680353 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:08.680335 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" Jul 24 15:52:08.680477 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:08.680371 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4" event={"ID":"5cab2266-896c-41fb-9365-d7cf84a47c4c","Type":"ContainerDied","Data":"08fa7679cbc7e541d22c1a238629612a5acbdbdbde2446b03a9fe2b538ab4741"} Jul 24 15:52:08.680477 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:08.680413 2569 scope.go:117] "RemoveContainer" containerID="41591a3620a934295b48b697247f603dae704c3e10fbea7758255d98609d332d" Jul 24 15:52:08.682632 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:08.682569 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qqczl" event={"ID":"0134774e-c656-4108-99a4-57714cc9e9f3","Type":"ContainerStarted","Data":"88c1cb923682a39050b544553c068a5a2959736f1db5b9572ee10604fec40af5"} Jul 24 15:52:08.683329 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:08.683299 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qqczl" Jul 24 15:52:08.696618 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:08.696574 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jcgj8" podStartSLOduration=2.777271507 podStartE2EDuration="5.696558565s" totalImagesPullingTime="2.919287058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:52:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:52:08.695088944 +0000 UTC m=+403.613059244" watchObservedRunningTime="2026-07-24 15:52:08.696558565 +0000 UTC m=+403.614528867" Jul 24 15:52:08.710563 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:08.710530 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4"] Jul 24 15:52:08.717064 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:08.717037 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-p42n4"] Jul 24 15:52:08.731038 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:08.730975 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qqczl" podStartSLOduration=2.834368516 podStartE2EDuration="5.73095549s" totalImagesPullingTime="2.896586974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:52:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:52:08.730390644 +0000 UTC m=+403.648360946" watchObservedRunningTime="2026-07-24 15:52:08.73095549 +0000 UTC m=+403.648925791" Jul 24 15:52:09.611648 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:09.611610 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-kp9qs" Jul 24 15:52:09.627568 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:09.627535 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cab2266-896c-41fb-9365-d7cf84a47c4c" path="/var/lib/kubelet/pods/5cab2266-896c-41fb-9365-d7cf84a47c4c/volumes" Jul 24 15:52:39.695333 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:39.695300 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jcgj8" Jul 24 15:52:40.697477 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:40.697442 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-qqczl" Jul 24 15:52:53.282461 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.282380 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q67b4"] Jul 24 15:52:53.282811 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.282763 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cab2266-896c-41fb-9365-d7cf84a47c4c" containerName="discovery" Jul 24 15:52:53.282811 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.282774 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cab2266-896c-41fb-9365-d7cf84a47c4c" containerName="discovery" Jul 24 15:52:53.282884 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.282845 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cab2266-896c-41fb-9365-d7cf84a47c4c" containerName="discovery" Jul 24 15:52:53.285876 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.285856 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q67b4" Jul 24 15:52:53.288528 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.288508 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-jsmrh\"" Jul 24 15:52:53.288637 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.288544 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 15:52:53.288637 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.288595 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 15:52:53.294246 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.294179 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q67b4"] Jul 24 15:52:53.331137 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.331103 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9fcg\" (UniqueName: \"kubernetes.io/projected/6062eea4-64a1-4e68-97a4-220c05987e23-kube-api-access-j9fcg\") pod \"kuadrant-operator-catalog-q67b4\" (UID: \"6062eea4-64a1-4e68-97a4-220c05987e23\") " pod="kuadrant-system/kuadrant-operator-catalog-q67b4" Jul 24 15:52:53.432475 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.432445 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9fcg\" (UniqueName: \"kubernetes.io/projected/6062eea4-64a1-4e68-97a4-220c05987e23-kube-api-access-j9fcg\") pod \"kuadrant-operator-catalog-q67b4\" (UID: \"6062eea4-64a1-4e68-97a4-220c05987e23\") " pod="kuadrant-system/kuadrant-operator-catalog-q67b4" Jul 24 15:52:53.440484 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.440454 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9fcg\" (UniqueName: \"kubernetes.io/projected/6062eea4-64a1-4e68-97a4-220c05987e23-kube-api-access-j9fcg\") pod \"kuadrant-operator-catalog-q67b4\" (UID: \"6062eea4-64a1-4e68-97a4-220c05987e23\") " pod="kuadrant-system/kuadrant-operator-catalog-q67b4" Jul 24 15:52:53.597022 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.596943 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q67b4" Jul 24 15:52:53.650396 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.650361 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q67b4"] Jul 24 15:52:53.728760 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.728735 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q67b4"] Jul 24 15:52:53.730297 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:52:53.730272 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6062eea4_64a1_4e68_97a4_220c05987e23.slice/crio-ff6b1435bab38485b7935dc4881cbea182679ecc8ba1a028047810fa37781ade WatchSource:0}: Error finding container ff6b1435bab38485b7935dc4881cbea182679ecc8ba1a028047810fa37781ade: Status 404 returned error can't find the container with id ff6b1435bab38485b7935dc4881cbea182679ecc8ba1a028047810fa37781ade Jul 24 15:52:53.857893 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.857815 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8dmw4"] Jul 24 15:52:53.862588 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.862573 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8dmw4" Jul 24 15:52:53.868855 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.868826 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8dmw4"] Jul 24 15:52:53.921220 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.921176 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q67b4" event={"ID":"6062eea4-64a1-4e68-97a4-220c05987e23","Type":"ContainerStarted","Data":"ff6b1435bab38485b7935dc4881cbea182679ecc8ba1a028047810fa37781ade"} Jul 24 15:52:53.936818 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:53.936787 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flpbz\" (UniqueName: \"kubernetes.io/projected/b0dafc08-22ce-4c05-9687-95dd771ea52c-kube-api-access-flpbz\") pod \"kuadrant-operator-catalog-8dmw4\" (UID: \"b0dafc08-22ce-4c05-9687-95dd771ea52c\") " pod="kuadrant-system/kuadrant-operator-catalog-8dmw4" Jul 24 15:52:54.037836 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:54.037800 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flpbz\" (UniqueName: \"kubernetes.io/projected/b0dafc08-22ce-4c05-9687-95dd771ea52c-kube-api-access-flpbz\") pod \"kuadrant-operator-catalog-8dmw4\" (UID: \"b0dafc08-22ce-4c05-9687-95dd771ea52c\") " pod="kuadrant-system/kuadrant-operator-catalog-8dmw4" Jul 24 15:52:54.047597 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:54.047555 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flpbz\" (UniqueName: \"kubernetes.io/projected/b0dafc08-22ce-4c05-9687-95dd771ea52c-kube-api-access-flpbz\") pod \"kuadrant-operator-catalog-8dmw4\" (UID: \"b0dafc08-22ce-4c05-9687-95dd771ea52c\") " pod="kuadrant-system/kuadrant-operator-catalog-8dmw4" Jul 24 15:52:54.174176 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:54.174140 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8dmw4" Jul 24 15:52:54.316757 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:54.316734 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8dmw4"] Jul 24 15:52:54.318335 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:52:54.318306 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0dafc08_22ce_4c05_9687_95dd771ea52c.slice/crio-c033b03d6a46ba829eeb1fa03cc1789910a340b04bdb90c2fa5cf2a0c9c3f4ae WatchSource:0}: Error finding container c033b03d6a46ba829eeb1fa03cc1789910a340b04bdb90c2fa5cf2a0c9c3f4ae: Status 404 returned error can't find the container with id c033b03d6a46ba829eeb1fa03cc1789910a340b04bdb90c2fa5cf2a0c9c3f4ae Jul 24 15:52:54.929878 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:54.929838 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8dmw4" event={"ID":"b0dafc08-22ce-4c05-9687-95dd771ea52c","Type":"ContainerStarted","Data":"c033b03d6a46ba829eeb1fa03cc1789910a340b04bdb90c2fa5cf2a0c9c3f4ae"} Jul 24 15:52:55.936993 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:55.936956 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q67b4" event={"ID":"6062eea4-64a1-4e68-97a4-220c05987e23","Type":"ContainerStarted","Data":"67850ad393ed8f332e37b817a127d27aeddd12b1b459ff6dacc135b16b3fc966"} Jul 24 15:52:55.937494 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:55.937013 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-q67b4" podUID="6062eea4-64a1-4e68-97a4-220c05987e23" containerName="registry-server" containerID="cri-o://67850ad393ed8f332e37b817a127d27aeddd12b1b459ff6dacc135b16b3fc966" gracePeriod=2 Jul 24 15:52:55.938717 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:55.938689 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8dmw4" event={"ID":"b0dafc08-22ce-4c05-9687-95dd771ea52c","Type":"ContainerStarted","Data":"e7d7b0649a245d7312dbeea6ad3f9625c81a8e14f9a34d4c8dc8f93c6cf5749a"} Jul 24 15:52:55.952634 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:55.952589 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-q67b4" podStartSLOduration=0.895369151 podStartE2EDuration="2.952572584s" totalImagesPullingTime="2.057203433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:52:55.951277701 +0000 UTC m=+450.869248001" watchObservedRunningTime="2026-07-24 15:52:55.952572584 +0000 UTC m=+450.870542883" Jul 24 15:52:55.966803 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:55.966747 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-8dmw4" podStartSLOduration=1.496638541 podStartE2EDuration="2.966724545s" totalImagesPullingTime="1.470086004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:52:55.965725279 +0000 UTC m=+450.883695602" watchObservedRunningTime="2026-07-24 15:52:55.966724545 +0000 UTC m=+450.884694846" Jul 24 15:52:56.194535 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:56.194512 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-q67b4_6062eea4-64a1-4e68-97a4-220c05987e23/registry-server/0.log" Jul 24 15:52:56.196817 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:56.196802 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q67b4" Jul 24 15:52:56.259249 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:56.259193 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9fcg\" (UniqueName: \"kubernetes.io/projected/6062eea4-64a1-4e68-97a4-220c05987e23-kube-api-access-j9fcg\") pod \"6062eea4-64a1-4e68-97a4-220c05987e23\" (UID: \"6062eea4-64a1-4e68-97a4-220c05987e23\") " Jul 24 15:52:56.261357 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:56.261333 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6062eea4-64a1-4e68-97a4-220c05987e23-kube-api-access-j9fcg" (OuterVolumeSpecName: "kube-api-access-j9fcg") pod "6062eea4-64a1-4e68-97a4-220c05987e23" (UID: "6062eea4-64a1-4e68-97a4-220c05987e23"). InnerVolumeSpecName "kube-api-access-j9fcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:52:56.359806 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:56.359713 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9fcg\" (UniqueName: \"kubernetes.io/projected/6062eea4-64a1-4e68-97a4-220c05987e23-kube-api-access-j9fcg\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:52:56.944554 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:56.944532 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-q67b4_6062eea4-64a1-4e68-97a4-220c05987e23/registry-server/0.log" Jul 24 15:52:56.944945 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:56.944568 2569 generic.go:356] "Generic (PLEG): container finished" podID="6062eea4-64a1-4e68-97a4-220c05987e23" containerID="67850ad393ed8f332e37b817a127d27aeddd12b1b459ff6dacc135b16b3fc966" exitCode=1 Jul 24 15:52:56.944945 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:56.944618 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-q67b4" Jul 24 15:52:56.944945 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:56.944657 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q67b4" event={"ID":"6062eea4-64a1-4e68-97a4-220c05987e23","Type":"ContainerDied","Data":"67850ad393ed8f332e37b817a127d27aeddd12b1b459ff6dacc135b16b3fc966"} Jul 24 15:52:56.944945 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:56.944706 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-q67b4" event={"ID":"6062eea4-64a1-4e68-97a4-220c05987e23","Type":"ContainerDied","Data":"ff6b1435bab38485b7935dc4881cbea182679ecc8ba1a028047810fa37781ade"} Jul 24 15:52:56.944945 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:56.944725 2569 scope.go:117] "RemoveContainer" containerID="67850ad393ed8f332e37b817a127d27aeddd12b1b459ff6dacc135b16b3fc966" Jul 24 15:52:56.954013 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:56.953998 2569 scope.go:117] "RemoveContainer" containerID="67850ad393ed8f332e37b817a127d27aeddd12b1b459ff6dacc135b16b3fc966" Jul 24 15:52:56.954284 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:52:56.954264 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67850ad393ed8f332e37b817a127d27aeddd12b1b459ff6dacc135b16b3fc966\": container with ID starting with 67850ad393ed8f332e37b817a127d27aeddd12b1b459ff6dacc135b16b3fc966 not found: ID does not exist" containerID="67850ad393ed8f332e37b817a127d27aeddd12b1b459ff6dacc135b16b3fc966" Jul 24 15:52:56.954353 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:56.954292 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67850ad393ed8f332e37b817a127d27aeddd12b1b459ff6dacc135b16b3fc966"} err="failed to get container status \"67850ad393ed8f332e37b817a127d27aeddd12b1b459ff6dacc135b16b3fc966\": rpc error: code = NotFound desc = could not find container \"67850ad393ed8f332e37b817a127d27aeddd12b1b459ff6dacc135b16b3fc966\": container with ID starting with 67850ad393ed8f332e37b817a127d27aeddd12b1b459ff6dacc135b16b3fc966 not found: ID does not exist" Jul 24 15:52:56.965004 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:56.964980 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q67b4"] Jul 24 15:52:56.968712 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:56.968693 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-q67b4"] Jul 24 15:52:57.627871 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:52:57.627841 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6062eea4-64a1-4e68-97a4-220c05987e23" path="/var/lib/kubelet/pods/6062eea4-64a1-4e68-97a4-220c05987e23/volumes" Jul 24 15:53:04.174392 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:04.174360 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-8dmw4" Jul 24 15:53:04.174831 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:04.174438 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-8dmw4" Jul 24 15:53:04.196347 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:04.196322 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-8dmw4" Jul 24 15:53:05.016830 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:05.016798 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-8dmw4" Jul 24 15:53:09.084728 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.084697 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw"] Jul 24 15:53:09.085166 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.085128 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6062eea4-64a1-4e68-97a4-220c05987e23" containerName="registry-server" Jul 24 15:53:09.085166 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.085141 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6062eea4-64a1-4e68-97a4-220c05987e23" containerName="registry-server" Jul 24 15:53:09.085300 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.085229 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6062eea4-64a1-4e68-97a4-220c05987e23" containerName="registry-server" Jul 24 15:53:09.090097 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.090077 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw" Jul 24 15:53:09.093411 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.093385 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bd4wh\"" Jul 24 15:53:09.094405 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.094380 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw"] Jul 24 15:53:09.169608 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.169571 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhbsw\" (UniqueName: \"kubernetes.io/projected/c74a98b2-dd57-4428-9518-c04852f7aca8-kube-api-access-vhbsw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw\" (UID: \"c74a98b2-dd57-4428-9518-c04852f7aca8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw" Jul 24 15:53:09.169790 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.169671 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c74a98b2-dd57-4428-9518-c04852f7aca8-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw\" (UID: \"c74a98b2-dd57-4428-9518-c04852f7aca8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw" Jul 24 15:53:09.169790 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.169702 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c74a98b2-dd57-4428-9518-c04852f7aca8-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw\" (UID: \"c74a98b2-dd57-4428-9518-c04852f7aca8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw" Jul 24 15:53:09.270543 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.270508 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhbsw\" (UniqueName: \"kubernetes.io/projected/c74a98b2-dd57-4428-9518-c04852f7aca8-kube-api-access-vhbsw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw\" (UID: \"c74a98b2-dd57-4428-9518-c04852f7aca8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw" Jul 24 15:53:09.270688 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.270601 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c74a98b2-dd57-4428-9518-c04852f7aca8-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw\" (UID: \"c74a98b2-dd57-4428-9518-c04852f7aca8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw" Jul 24 15:53:09.270688 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.270636 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c74a98b2-dd57-4428-9518-c04852f7aca8-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw\" (UID: \"c74a98b2-dd57-4428-9518-c04852f7aca8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw" Jul 24 15:53:09.271052 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.271033 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c74a98b2-dd57-4428-9518-c04852f7aca8-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw\" (UID: \"c74a98b2-dd57-4428-9518-c04852f7aca8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw" Jul 24 15:53:09.271090 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.271069 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c74a98b2-dd57-4428-9518-c04852f7aca8-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw\" (UID: \"c74a98b2-dd57-4428-9518-c04852f7aca8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw" Jul 24 15:53:09.278835 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.278807 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhbsw\" (UniqueName: \"kubernetes.io/projected/c74a98b2-dd57-4428-9518-c04852f7aca8-kube-api-access-vhbsw\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw\" (UID: \"c74a98b2-dd57-4428-9518-c04852f7aca8\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw" Jul 24 15:53:09.400520 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.400486 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw" Jul 24 15:53:09.535793 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.535766 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw"] Jul 24 15:53:09.537201 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:53:09.537177 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc74a98b2_dd57_4428_9518_c04852f7aca8.slice/crio-cd1e10a47bf9250d31ba96304cf54201539c149564ff6eb75b1f6fc09ffa141b WatchSource:0}: Error finding container cd1e10a47bf9250d31ba96304cf54201539c149564ff6eb75b1f6fc09ffa141b: Status 404 returned error can't find the container with id cd1e10a47bf9250d31ba96304cf54201539c149564ff6eb75b1f6fc09ffa141b Jul 24 15:53:09.686889 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.686854 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s"] Jul 24 15:53:09.690609 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.690591 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s" Jul 24 15:53:09.697954 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.697927 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s"] Jul 24 15:53:09.775463 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.775426 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/373c13c5-cd6e-4565-b34e-768266c0ec26-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s\" (UID: \"373c13c5-cd6e-4565-b34e-768266c0ec26\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s" Jul 24 15:53:09.775636 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.775472 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/373c13c5-cd6e-4565-b34e-768266c0ec26-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s\" (UID: \"373c13c5-cd6e-4565-b34e-768266c0ec26\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s" Jul 24 15:53:09.775636 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.775537 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmq7g\" (UniqueName: \"kubernetes.io/projected/373c13c5-cd6e-4565-b34e-768266c0ec26-kube-api-access-dmq7g\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s\" (UID: \"373c13c5-cd6e-4565-b34e-768266c0ec26\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s" Jul 24 15:53:09.876393 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.876340 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmq7g\" (UniqueName: \"kubernetes.io/projected/373c13c5-cd6e-4565-b34e-768266c0ec26-kube-api-access-dmq7g\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s\" (UID: \"373c13c5-cd6e-4565-b34e-768266c0ec26\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s" Jul 24 15:53:09.876573 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.876426 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/373c13c5-cd6e-4565-b34e-768266c0ec26-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s\" (UID: \"373c13c5-cd6e-4565-b34e-768266c0ec26\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s" Jul 24 15:53:09.876573 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.876456 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/373c13c5-cd6e-4565-b34e-768266c0ec26-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s\" (UID: \"373c13c5-cd6e-4565-b34e-768266c0ec26\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s" Jul 24 15:53:09.876763 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.876749 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/373c13c5-cd6e-4565-b34e-768266c0ec26-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s\" (UID: \"373c13c5-cd6e-4565-b34e-768266c0ec26\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s" Jul 24 15:53:09.876858 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.876841 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/373c13c5-cd6e-4565-b34e-768266c0ec26-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s\" (UID: \"373c13c5-cd6e-4565-b34e-768266c0ec26\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s" Jul 24 15:53:09.885015 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:09.884997 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmq7g\" (UniqueName: \"kubernetes.io/projected/373c13c5-cd6e-4565-b34e-768266c0ec26-kube-api-access-dmq7g\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s\" (UID: \"373c13c5-cd6e-4565-b34e-768266c0ec26\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s" Jul 24 15:53:10.013460 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.013376 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s" Jul 24 15:53:10.023349 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.023320 2569 generic.go:356] "Generic (PLEG): container finished" podID="c74a98b2-dd57-4428-9518-c04852f7aca8" containerID="71ca253ff9856138dfe6a6ce902af9fbcc69328309dcb4916e8dee02cf0130c2" exitCode=0 Jul 24 15:53:10.023453 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.023362 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw" event={"ID":"c74a98b2-dd57-4428-9518-c04852f7aca8","Type":"ContainerDied","Data":"71ca253ff9856138dfe6a6ce902af9fbcc69328309dcb4916e8dee02cf0130c2"} Jul 24 15:53:10.023453 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.023405 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw" event={"ID":"c74a98b2-dd57-4428-9518-c04852f7aca8","Type":"ContainerStarted","Data":"cd1e10a47bf9250d31ba96304cf54201539c149564ff6eb75b1f6fc09ffa141b"} Jul 24 15:53:10.084873 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.084844 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd"] Jul 24 15:53:10.089725 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.089705 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd" Jul 24 15:53:10.094518 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.094498 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd"] Jul 24 15:53:10.152522 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.152495 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s"] Jul 24 15:53:10.153842 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:53:10.153816 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373c13c5_cd6e_4565_b34e_768266c0ec26.slice/crio-d7d9c9f77d6f398181fc79b67855624840e44a23908dde826796f0c25f3f8e0c WatchSource:0}: Error finding container d7d9c9f77d6f398181fc79b67855624840e44a23908dde826796f0c25f3f8e0c: Status 404 returned error can't find the container with id d7d9c9f77d6f398181fc79b67855624840e44a23908dde826796f0c25f3f8e0c Jul 24 15:53:10.180177 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.180149 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42a19dbb-d098-4980-930b-f9440ecca8ae-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd\" (UID: \"42a19dbb-d098-4980-930b-f9440ecca8ae\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd" Jul 24 15:53:10.180297 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.180182 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42a19dbb-d098-4980-930b-f9440ecca8ae-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd\" (UID: \"42a19dbb-d098-4980-930b-f9440ecca8ae\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd" Jul 24 15:53:10.180297 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.180202 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp9c2\" (UniqueName: \"kubernetes.io/projected/42a19dbb-d098-4980-930b-f9440ecca8ae-kube-api-access-mp9c2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd\" (UID: \"42a19dbb-d098-4980-930b-f9440ecca8ae\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd" Jul 24 15:53:10.280988 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.280909 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42a19dbb-d098-4980-930b-f9440ecca8ae-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd\" (UID: \"42a19dbb-d098-4980-930b-f9440ecca8ae\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd" Jul 24 15:53:10.280988 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.280946 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42a19dbb-d098-4980-930b-f9440ecca8ae-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd\" (UID: \"42a19dbb-d098-4980-930b-f9440ecca8ae\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd" Jul 24 15:53:10.280988 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.280965 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp9c2\" (UniqueName: \"kubernetes.io/projected/42a19dbb-d098-4980-930b-f9440ecca8ae-kube-api-access-mp9c2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd\" (UID: \"42a19dbb-d098-4980-930b-f9440ecca8ae\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd" Jul 24 15:53:10.281413 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.281392 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42a19dbb-d098-4980-930b-f9440ecca8ae-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd\" (UID: \"42a19dbb-d098-4980-930b-f9440ecca8ae\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd" Jul 24 15:53:10.281413 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.281404 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42a19dbb-d098-4980-930b-f9440ecca8ae-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd\" (UID: \"42a19dbb-d098-4980-930b-f9440ecca8ae\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd" Jul 24 15:53:10.289855 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.289832 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp9c2\" (UniqueName: \"kubernetes.io/projected/42a19dbb-d098-4980-930b-f9440ecca8ae-kube-api-access-mp9c2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd\" (UID: \"42a19dbb-d098-4980-930b-f9440ecca8ae\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd" Jul 24 15:53:10.400632 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.400600 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd" Jul 24 15:53:10.490020 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.489991 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b"] Jul 24 15:53:10.495092 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.495068 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b" Jul 24 15:53:10.499916 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.499893 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b"] Jul 24 15:53:10.561043 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.561016 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd"] Jul 24 15:53:10.562863 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:53:10.562831 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a19dbb_d098_4980_930b_f9440ecca8ae.slice/crio-ee3fe5fec78d119025269114fc575be95ea7c97bc9415f6fa4e2b580bc93b681 WatchSource:0}: Error finding container ee3fe5fec78d119025269114fc575be95ea7c97bc9415f6fa4e2b580bc93b681: Status 404 returned error can't find the container with id ee3fe5fec78d119025269114fc575be95ea7c97bc9415f6fa4e2b580bc93b681 Jul 24 15:53:10.583641 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.583617 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01c8ce33-7591-4b16-a830-114abddd3507-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b\" (UID: \"01c8ce33-7591-4b16-a830-114abddd3507\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b" Jul 24 15:53:10.583822 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.583663 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpv95\" (UniqueName: \"kubernetes.io/projected/01c8ce33-7591-4b16-a830-114abddd3507-kube-api-access-mpv95\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b\" (UID: \"01c8ce33-7591-4b16-a830-114abddd3507\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b" Jul 24 15:53:10.583822 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.583690 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01c8ce33-7591-4b16-a830-114abddd3507-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b\" (UID: \"01c8ce33-7591-4b16-a830-114abddd3507\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b" Jul 24 15:53:10.684372 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.684332 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01c8ce33-7591-4b16-a830-114abddd3507-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b\" (UID: \"01c8ce33-7591-4b16-a830-114abddd3507\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b" Jul 24 15:53:10.684525 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.684432 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01c8ce33-7591-4b16-a830-114abddd3507-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b\" (UID: \"01c8ce33-7591-4b16-a830-114abddd3507\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b" Jul 24 15:53:10.684525 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.684473 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpv95\" (UniqueName: \"kubernetes.io/projected/01c8ce33-7591-4b16-a830-114abddd3507-kube-api-access-mpv95\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b\" (UID: \"01c8ce33-7591-4b16-a830-114abddd3507\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b" Jul 24 15:53:10.684747 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.684722 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01c8ce33-7591-4b16-a830-114abddd3507-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b\" (UID: \"01c8ce33-7591-4b16-a830-114abddd3507\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b" Jul 24 15:53:10.684784 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.684744 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01c8ce33-7591-4b16-a830-114abddd3507-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b\" (UID: \"01c8ce33-7591-4b16-a830-114abddd3507\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b" Jul 24 15:53:10.692928 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.692907 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpv95\" (UniqueName: \"kubernetes.io/projected/01c8ce33-7591-4b16-a830-114abddd3507-kube-api-access-mpv95\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b\" (UID: \"01c8ce33-7591-4b16-a830-114abddd3507\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b" Jul 24 15:53:10.806881 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.806849 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b" Jul 24 15:53:10.964543 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:10.964509 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b"] Jul 24 15:53:10.990123 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:53:10.990091 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c8ce33_7591_4b16_a830_114abddd3507.slice/crio-c0de047fc0c59564778ec09b50cfeec38863933192a205916a70837378307e10 WatchSource:0}: Error finding container c0de047fc0c59564778ec09b50cfeec38863933192a205916a70837378307e10: Status 404 returned error can't find the container with id c0de047fc0c59564778ec09b50cfeec38863933192a205916a70837378307e10 Jul 24 15:53:11.030739 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:11.030711 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b" event={"ID":"01c8ce33-7591-4b16-a830-114abddd3507","Type":"ContainerStarted","Data":"c0de047fc0c59564778ec09b50cfeec38863933192a205916a70837378307e10"} Jul 24 15:53:11.032328 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:11.032300 2569 generic.go:356] "Generic (PLEG): container finished" podID="373c13c5-cd6e-4565-b34e-768266c0ec26" containerID="c84d5660ad13a93e346340497f126fbc707b663f02ffa1ab767c9f8faab0dbb6" exitCode=0 Jul 24 15:53:11.032440 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:11.032374 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s" event={"ID":"373c13c5-cd6e-4565-b34e-768266c0ec26","Type":"ContainerDied","Data":"c84d5660ad13a93e346340497f126fbc707b663f02ffa1ab767c9f8faab0dbb6"} Jul 24 15:53:11.032440 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:11.032422 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s" event={"ID":"373c13c5-cd6e-4565-b34e-768266c0ec26","Type":"ContainerStarted","Data":"d7d9c9f77d6f398181fc79b67855624840e44a23908dde826796f0c25f3f8e0c"} Jul 24 15:53:11.034477 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:11.034455 2569 generic.go:356] "Generic (PLEG): container finished" podID="c74a98b2-dd57-4428-9518-c04852f7aca8" containerID="0a20d6b10ff5d6cf5beed885fbd34557c100472e5776a8d7b67a10dcdbc0b1f2" exitCode=0 Jul 24 15:53:11.034582 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:11.034513 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw" event={"ID":"c74a98b2-dd57-4428-9518-c04852f7aca8","Type":"ContainerDied","Data":"0a20d6b10ff5d6cf5beed885fbd34557c100472e5776a8d7b67a10dcdbc0b1f2"} Jul 24 15:53:11.036561 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:11.036540 2569 generic.go:356] "Generic (PLEG): container finished" podID="42a19dbb-d098-4980-930b-f9440ecca8ae" containerID="233efb479dfe069364d243c76b80790cff1ef7887b59706d82a477ff74a1d988" exitCode=0 Jul 24 15:53:11.036723 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:11.036590 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd" event={"ID":"42a19dbb-d098-4980-930b-f9440ecca8ae","Type":"ContainerDied","Data":"233efb479dfe069364d243c76b80790cff1ef7887b59706d82a477ff74a1d988"} Jul 24 15:53:11.036723 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:11.036625 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd" event={"ID":"42a19dbb-d098-4980-930b-f9440ecca8ae","Type":"ContainerStarted","Data":"ee3fe5fec78d119025269114fc575be95ea7c97bc9415f6fa4e2b580bc93b681"} Jul 24 15:53:12.046039 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:12.046005 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s" event={"ID":"373c13c5-cd6e-4565-b34e-768266c0ec26","Type":"ContainerStarted","Data":"81c56579fd7b02ee078ff34181c443ff7a539aea25b8df1388de6ead231f6569"} Jul 24 15:53:12.048198 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:12.048173 2569 generic.go:356] "Generic (PLEG): container finished" podID="c74a98b2-dd57-4428-9518-c04852f7aca8" containerID="f7c91b1e830e433d1f194363256f1d131d3798d68dd6bf5e77255b28e270900a" exitCode=0 Jul 24 15:53:12.048367 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:12.048266 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw" event={"ID":"c74a98b2-dd57-4428-9518-c04852f7aca8","Type":"ContainerDied","Data":"f7c91b1e830e433d1f194363256f1d131d3798d68dd6bf5e77255b28e270900a"} Jul 24 15:53:12.058456 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:12.058431 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd" event={"ID":"42a19dbb-d098-4980-930b-f9440ecca8ae","Type":"ContainerStarted","Data":"6ec02c73ef4d946a3a393bcc375c89c0254b46ec8e95306a78bb768c413371ca"} Jul 24 15:53:12.059980 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:12.059959 2569 generic.go:356] "Generic (PLEG): container finished" podID="01c8ce33-7591-4b16-a830-114abddd3507" containerID="f9b9b744201e4146dd4ad1aeffe6661982c49e87869fcd2048f958ab851647ae" exitCode=0 Jul 24 15:53:12.060080 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:12.060004 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b" event={"ID":"01c8ce33-7591-4b16-a830-114abddd3507","Type":"ContainerDied","Data":"f9b9b744201e4146dd4ad1aeffe6661982c49e87869fcd2048f958ab851647ae"} Jul 24 15:53:13.068185 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.068091 2569 generic.go:356] "Generic (PLEG): container finished" podID="01c8ce33-7591-4b16-a830-114abddd3507" containerID="81945d5025c011f3c31834d190f3c70c129e1ec0aafd4d7260eed2aad3c745cf" exitCode=0 Jul 24 15:53:13.068637 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.068242 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b" event={"ID":"01c8ce33-7591-4b16-a830-114abddd3507","Type":"ContainerDied","Data":"81945d5025c011f3c31834d190f3c70c129e1ec0aafd4d7260eed2aad3c745cf"} Jul 24 15:53:13.070751 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.070727 2569 generic.go:356] "Generic (PLEG): container finished" podID="373c13c5-cd6e-4565-b34e-768266c0ec26" containerID="81c56579fd7b02ee078ff34181c443ff7a539aea25b8df1388de6ead231f6569" exitCode=0 Jul 24 15:53:13.070751 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.070748 2569 generic.go:356] "Generic (PLEG): container finished" podID="373c13c5-cd6e-4565-b34e-768266c0ec26" containerID="3814c0e343d8687f1ac02c074d0ac732669782a21769527a57a718503c525b69" exitCode=0 Jul 24 15:53:13.071234 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.070816 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s" event={"ID":"373c13c5-cd6e-4565-b34e-768266c0ec26","Type":"ContainerDied","Data":"81c56579fd7b02ee078ff34181c443ff7a539aea25b8df1388de6ead231f6569"} Jul 24 15:53:13.071234 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.070868 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s" event={"ID":"373c13c5-cd6e-4565-b34e-768266c0ec26","Type":"ContainerDied","Data":"3814c0e343d8687f1ac02c074d0ac732669782a21769527a57a718503c525b69"} Jul 24 15:53:13.073058 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.073035 2569 generic.go:356] "Generic (PLEG): container finished" podID="42a19dbb-d098-4980-930b-f9440ecca8ae" containerID="6ec02c73ef4d946a3a393bcc375c89c0254b46ec8e95306a78bb768c413371ca" exitCode=0 Jul 24 15:53:13.073159 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.073059 2569 generic.go:356] "Generic (PLEG): container finished" podID="42a19dbb-d098-4980-930b-f9440ecca8ae" containerID="e9a8a08d4295b6b02d7bba6122b5d91fea7a1c36ce0d44a5e9d158c8c8aeeea2" exitCode=0 Jul 24 15:53:13.073159 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.073111 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd" event={"ID":"42a19dbb-d098-4980-930b-f9440ecca8ae","Type":"ContainerDied","Data":"6ec02c73ef4d946a3a393bcc375c89c0254b46ec8e95306a78bb768c413371ca"} Jul 24 15:53:13.073159 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.073148 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd" event={"ID":"42a19dbb-d098-4980-930b-f9440ecca8ae","Type":"ContainerDied","Data":"e9a8a08d4295b6b02d7bba6122b5d91fea7a1c36ce0d44a5e9d158c8c8aeeea2"} Jul 24 15:53:13.285663 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.285636 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw" Jul 24 15:53:13.409392 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.409352 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhbsw\" (UniqueName: \"kubernetes.io/projected/c74a98b2-dd57-4428-9518-c04852f7aca8-kube-api-access-vhbsw\") pod \"c74a98b2-dd57-4428-9518-c04852f7aca8\" (UID: \"c74a98b2-dd57-4428-9518-c04852f7aca8\") " Jul 24 15:53:13.409570 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.409472 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c74a98b2-dd57-4428-9518-c04852f7aca8-util\") pod \"c74a98b2-dd57-4428-9518-c04852f7aca8\" (UID: \"c74a98b2-dd57-4428-9518-c04852f7aca8\") " Jul 24 15:53:13.409570 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.409497 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c74a98b2-dd57-4428-9518-c04852f7aca8-bundle\") pod \"c74a98b2-dd57-4428-9518-c04852f7aca8\" (UID: \"c74a98b2-dd57-4428-9518-c04852f7aca8\") " Jul 24 15:53:13.410170 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.410138 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74a98b2-dd57-4428-9518-c04852f7aca8-bundle" (OuterVolumeSpecName: "bundle") pod "c74a98b2-dd57-4428-9518-c04852f7aca8" (UID: "c74a98b2-dd57-4428-9518-c04852f7aca8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:53:13.411523 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.411502 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c74a98b2-dd57-4428-9518-c04852f7aca8-kube-api-access-vhbsw" (OuterVolumeSpecName: "kube-api-access-vhbsw") pod "c74a98b2-dd57-4428-9518-c04852f7aca8" (UID: "c74a98b2-dd57-4428-9518-c04852f7aca8"). InnerVolumeSpecName "kube-api-access-vhbsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:53:13.415176 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.415152 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c74a98b2-dd57-4428-9518-c04852f7aca8-util" (OuterVolumeSpecName: "util") pod "c74a98b2-dd57-4428-9518-c04852f7aca8" (UID: "c74a98b2-dd57-4428-9518-c04852f7aca8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:53:13.511118 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.511072 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c74a98b2-dd57-4428-9518-c04852f7aca8-util\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:53:13.511118 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.511112 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c74a98b2-dd57-4428-9518-c04852f7aca8-bundle\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:53:13.511118 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:13.511122 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhbsw\" (UniqueName: \"kubernetes.io/projected/c74a98b2-dd57-4428-9518-c04852f7aca8-kube-api-access-vhbsw\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:53:14.079828 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.079798 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw" Jul 24 15:53:14.080259 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.079826 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw" event={"ID":"c74a98b2-dd57-4428-9518-c04852f7aca8","Type":"ContainerDied","Data":"cd1e10a47bf9250d31ba96304cf54201539c149564ff6eb75b1f6fc09ffa141b"} Jul 24 15:53:14.080259 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.079876 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd1e10a47bf9250d31ba96304cf54201539c149564ff6eb75b1f6fc09ffa141b" Jul 24 15:53:14.082228 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.082189 2569 generic.go:356] "Generic (PLEG): container finished" podID="01c8ce33-7591-4b16-a830-114abddd3507" containerID="0eab7d4e19c1f1c90138d4ab64141f8c59687fdf689f6ec4228e25eccb52bb02" exitCode=0 Jul 24 15:53:14.082351 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.082239 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b" event={"ID":"01c8ce33-7591-4b16-a830-114abddd3507","Type":"ContainerDied","Data":"0eab7d4e19c1f1c90138d4ab64141f8c59687fdf689f6ec4228e25eccb52bb02"} Jul 24 15:53:14.243918 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.243890 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd" Jul 24 15:53:14.245915 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.245898 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s" Jul 24 15:53:14.318405 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.318372 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/373c13c5-cd6e-4565-b34e-768266c0ec26-bundle\") pod \"373c13c5-cd6e-4565-b34e-768266c0ec26\" (UID: \"373c13c5-cd6e-4565-b34e-768266c0ec26\") " Jul 24 15:53:14.318598 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.318434 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/373c13c5-cd6e-4565-b34e-768266c0ec26-util\") pod \"373c13c5-cd6e-4565-b34e-768266c0ec26\" (UID: \"373c13c5-cd6e-4565-b34e-768266c0ec26\") " Jul 24 15:53:14.318598 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.318478 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42a19dbb-d098-4980-930b-f9440ecca8ae-util\") pod \"42a19dbb-d098-4980-930b-f9440ecca8ae\" (UID: \"42a19dbb-d098-4980-930b-f9440ecca8ae\") " Jul 24 15:53:14.318598 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.318530 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp9c2\" (UniqueName: \"kubernetes.io/projected/42a19dbb-d098-4980-930b-f9440ecca8ae-kube-api-access-mp9c2\") pod \"42a19dbb-d098-4980-930b-f9440ecca8ae\" (UID: \"42a19dbb-d098-4980-930b-f9440ecca8ae\") " Jul 24 15:53:14.318598 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.318564 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmq7g\" (UniqueName: \"kubernetes.io/projected/373c13c5-cd6e-4565-b34e-768266c0ec26-kube-api-access-dmq7g\") pod \"373c13c5-cd6e-4565-b34e-768266c0ec26\" (UID: \"373c13c5-cd6e-4565-b34e-768266c0ec26\") " Jul 24 15:53:14.318598 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.318591 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42a19dbb-d098-4980-930b-f9440ecca8ae-bundle\") pod \"42a19dbb-d098-4980-930b-f9440ecca8ae\" (UID: \"42a19dbb-d098-4980-930b-f9440ecca8ae\") " Jul 24 15:53:14.319010 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.318979 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373c13c5-cd6e-4565-b34e-768266c0ec26-bundle" (OuterVolumeSpecName: "bundle") pod "373c13c5-cd6e-4565-b34e-768266c0ec26" (UID: "373c13c5-cd6e-4565-b34e-768266c0ec26"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:53:14.319259 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.319232 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a19dbb-d098-4980-930b-f9440ecca8ae-bundle" (OuterVolumeSpecName: "bundle") pod "42a19dbb-d098-4980-930b-f9440ecca8ae" (UID: "42a19dbb-d098-4980-930b-f9440ecca8ae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:53:14.320870 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.320848 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373c13c5-cd6e-4565-b34e-768266c0ec26-kube-api-access-dmq7g" (OuterVolumeSpecName: "kube-api-access-dmq7g") pod "373c13c5-cd6e-4565-b34e-768266c0ec26" (UID: "373c13c5-cd6e-4565-b34e-768266c0ec26"). InnerVolumeSpecName "kube-api-access-dmq7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:53:14.321049 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.321029 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a19dbb-d098-4980-930b-f9440ecca8ae-kube-api-access-mp9c2" (OuterVolumeSpecName: "kube-api-access-mp9c2") pod "42a19dbb-d098-4980-930b-f9440ecca8ae" (UID: "42a19dbb-d098-4980-930b-f9440ecca8ae"). InnerVolumeSpecName "kube-api-access-mp9c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:53:14.324842 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.324822 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/373c13c5-cd6e-4565-b34e-768266c0ec26-util" (OuterVolumeSpecName: "util") pod "373c13c5-cd6e-4565-b34e-768266c0ec26" (UID: "373c13c5-cd6e-4565-b34e-768266c0ec26"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:53:14.326592 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.326573 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a19dbb-d098-4980-930b-f9440ecca8ae-util" (OuterVolumeSpecName: "util") pod "42a19dbb-d098-4980-930b-f9440ecca8ae" (UID: "42a19dbb-d098-4980-930b-f9440ecca8ae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:53:14.419552 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.419524 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/373c13c5-cd6e-4565-b34e-768266c0ec26-bundle\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:53:14.419552 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.419550 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/373c13c5-cd6e-4565-b34e-768266c0ec26-util\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:53:14.419552 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.419558 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42a19dbb-d098-4980-930b-f9440ecca8ae-util\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:53:14.419789 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.419567 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp9c2\" (UniqueName: \"kubernetes.io/projected/42a19dbb-d098-4980-930b-f9440ecca8ae-kube-api-access-mp9c2\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:53:14.419789 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.419576 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmq7g\" (UniqueName: \"kubernetes.io/projected/373c13c5-cd6e-4565-b34e-768266c0ec26-kube-api-access-dmq7g\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:53:14.419789 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:14.419585 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42a19dbb-d098-4980-930b-f9440ecca8ae-bundle\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:53:15.089751 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:15.089718 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s" Jul 24 15:53:15.090167 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:15.089718 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s" event={"ID":"373c13c5-cd6e-4565-b34e-768266c0ec26","Type":"ContainerDied","Data":"d7d9c9f77d6f398181fc79b67855624840e44a23908dde826796f0c25f3f8e0c"} Jul 24 15:53:15.090167 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:15.089846 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7d9c9f77d6f398181fc79b67855624840e44a23908dde826796f0c25f3f8e0c" Jul 24 15:53:15.091662 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:15.091643 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd" Jul 24 15:53:15.091780 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:15.091678 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd" event={"ID":"42a19dbb-d098-4980-930b-f9440ecca8ae","Type":"ContainerDied","Data":"ee3fe5fec78d119025269114fc575be95ea7c97bc9415f6fa4e2b580bc93b681"} Jul 24 15:53:15.091780 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:15.091714 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee3fe5fec78d119025269114fc575be95ea7c97bc9415f6fa4e2b580bc93b681" Jul 24 15:53:15.224357 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:15.224333 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b" Jul 24 15:53:15.325936 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:15.325905 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01c8ce33-7591-4b16-a830-114abddd3507-util\") pod \"01c8ce33-7591-4b16-a830-114abddd3507\" (UID: \"01c8ce33-7591-4b16-a830-114abddd3507\") " Jul 24 15:53:15.326094 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:15.325986 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01c8ce33-7591-4b16-a830-114abddd3507-bundle\") pod \"01c8ce33-7591-4b16-a830-114abddd3507\" (UID: \"01c8ce33-7591-4b16-a830-114abddd3507\") " Jul 24 15:53:15.326094 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:15.326021 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpv95\" (UniqueName: \"kubernetes.io/projected/01c8ce33-7591-4b16-a830-114abddd3507-kube-api-access-mpv95\") pod \"01c8ce33-7591-4b16-a830-114abddd3507\" (UID: \"01c8ce33-7591-4b16-a830-114abddd3507\") " Jul 24 15:53:15.331116 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:15.327115 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c8ce33-7591-4b16-a830-114abddd3507-bundle" (OuterVolumeSpecName: "bundle") pod "01c8ce33-7591-4b16-a830-114abddd3507" (UID: "01c8ce33-7591-4b16-a830-114abddd3507"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:53:15.333071 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:15.333016 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c8ce33-7591-4b16-a830-114abddd3507-kube-api-access-mpv95" (OuterVolumeSpecName: "kube-api-access-mpv95") pod "01c8ce33-7591-4b16-a830-114abddd3507" (UID: "01c8ce33-7591-4b16-a830-114abddd3507"). InnerVolumeSpecName "kube-api-access-mpv95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:53:15.334503 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:15.334450 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c8ce33-7591-4b16-a830-114abddd3507-util" (OuterVolumeSpecName: "util") pod "01c8ce33-7591-4b16-a830-114abddd3507" (UID: "01c8ce33-7591-4b16-a830-114abddd3507"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:53:15.427024 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:15.426991 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01c8ce33-7591-4b16-a830-114abddd3507-util\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:53:15.427024 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:15.427022 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01c8ce33-7591-4b16-a830-114abddd3507-bundle\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:53:15.427024 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:15.427032 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpv95\" (UniqueName: \"kubernetes.io/projected/01c8ce33-7591-4b16-a830-114abddd3507-kube-api-access-mpv95\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:53:16.100292 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:16.100251 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b" event={"ID":"01c8ce33-7591-4b16-a830-114abddd3507","Type":"ContainerDied","Data":"c0de047fc0c59564778ec09b50cfeec38863933192a205916a70837378307e10"} Jul 24 15:53:16.100690 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:16.100312 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0de047fc0c59564778ec09b50cfeec38863933192a205916a70837378307e10" Jul 24 15:53:16.100690 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:16.100268 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b" Jul 24 15:53:25.593023 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.592987 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-65kzp"] Jul 24 15:53:25.593413 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593398 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a19dbb-d098-4980-930b-f9440ecca8ae" containerName="pull" Jul 24 15:53:25.593413 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593414 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a19dbb-d098-4980-930b-f9440ecca8ae" containerName="pull" Jul 24 15:53:25.593487 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593425 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74a98b2-dd57-4428-9518-c04852f7aca8" containerName="extract" Jul 24 15:53:25.593487 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593431 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74a98b2-dd57-4428-9518-c04852f7aca8" containerName="extract" Jul 24 15:53:25.593487 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593439 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373c13c5-cd6e-4565-b34e-768266c0ec26" containerName="pull" Jul 24 15:53:25.593487 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593444 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="373c13c5-cd6e-4565-b34e-768266c0ec26" containerName="pull" Jul 24 15:53:25.593487 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593458 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a19dbb-d098-4980-930b-f9440ecca8ae" containerName="util" Jul 24 15:53:25.593487 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593464 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a19dbb-d098-4980-930b-f9440ecca8ae" containerName="util" Jul 24 15:53:25.593487 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593473 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74a98b2-dd57-4428-9518-c04852f7aca8" containerName="pull" Jul 24 15:53:25.593487 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593478 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74a98b2-dd57-4428-9518-c04852f7aca8" containerName="pull" Jul 24 15:53:25.593487 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593485 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373c13c5-cd6e-4565-b34e-768266c0ec26" containerName="extract" Jul 24 15:53:25.593487 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593491 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="373c13c5-cd6e-4565-b34e-768266c0ec26" containerName="extract" Jul 24 15:53:25.593769 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593497 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373c13c5-cd6e-4565-b34e-768266c0ec26" containerName="util" Jul 24 15:53:25.593769 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593502 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="373c13c5-cd6e-4565-b34e-768266c0ec26" containerName="util" Jul 24 15:53:25.593769 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593508 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01c8ce33-7591-4b16-a830-114abddd3507" containerName="util" Jul 24 15:53:25.593769 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593513 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="01c8ce33-7591-4b16-a830-114abddd3507" containerName="util" Jul 24 15:53:25.593769 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593522 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01c8ce33-7591-4b16-a830-114abddd3507" containerName="pull" Jul 24 15:53:25.593769 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593526 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="01c8ce33-7591-4b16-a830-114abddd3507" containerName="pull" Jul 24 15:53:25.593769 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593536 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c74a98b2-dd57-4428-9518-c04852f7aca8" containerName="util" Jul 24 15:53:25.593769 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593541 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c74a98b2-dd57-4428-9518-c04852f7aca8" containerName="util" Jul 24 15:53:25.593769 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593550 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a19dbb-d098-4980-930b-f9440ecca8ae" containerName="extract" Jul 24 15:53:25.593769 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593554 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a19dbb-d098-4980-930b-f9440ecca8ae" containerName="extract" Jul 24 15:53:25.593769 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593560 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01c8ce33-7591-4b16-a830-114abddd3507" containerName="extract" Jul 24 15:53:25.593769 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593565 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="01c8ce33-7591-4b16-a830-114abddd3507" containerName="extract" Jul 24 15:53:25.593769 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593618 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="42a19dbb-d098-4980-930b-f9440ecca8ae" containerName="extract" Jul 24 15:53:25.593769 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593626 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="373c13c5-cd6e-4565-b34e-768266c0ec26" containerName="extract" Jul 24 15:53:25.593769 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593632 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="01c8ce33-7591-4b16-a830-114abddd3507" containerName="extract" Jul 24 15:53:25.593769 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.593636 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c74a98b2-dd57-4428-9518-c04852f7aca8" containerName="extract" Jul 24 15:53:25.597920 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.597900 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-65kzp" Jul 24 15:53:25.602011 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.601811 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 24 15:53:25.602011 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.601853 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-lfdb7\"" Jul 24 15:53:25.613357 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.613325 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-65kzp"] Jul 24 15:53:25.720533 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.720500 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bljxd\" (UniqueName: \"kubernetes.io/projected/3db27a28-ad12-4c20-b8f6-d83a6ec42a09-kube-api-access-bljxd\") pod \"dns-operator-controller-manager-864dd78c4c-65kzp\" (UID: \"3db27a28-ad12-4c20-b8f6-d83a6ec42a09\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-65kzp" Jul 24 15:53:25.821787 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.821711 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bljxd\" (UniqueName: \"kubernetes.io/projected/3db27a28-ad12-4c20-b8f6-d83a6ec42a09-kube-api-access-bljxd\") pod \"dns-operator-controller-manager-864dd78c4c-65kzp\" (UID: \"3db27a28-ad12-4c20-b8f6-d83a6ec42a09\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-65kzp" Jul 24 15:53:25.829582 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.829555 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bljxd\" (UniqueName: \"kubernetes.io/projected/3db27a28-ad12-4c20-b8f6-d83a6ec42a09-kube-api-access-bljxd\") pod \"dns-operator-controller-manager-864dd78c4c-65kzp\" (UID: \"3db27a28-ad12-4c20-b8f6-d83a6ec42a09\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-65kzp" Jul 24 15:53:25.914501 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:25.914466 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-65kzp" Jul 24 15:53:26.053265 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.053239 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-65kzp"] Jul 24 15:53:26.054690 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:53:26.054660 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db27a28_ad12_4c20_b8f6_d83a6ec42a09.slice/crio-0256d300936b721b6d0dc8115e30270c93d51431e4bbce2b76b2323fedd72cdf WatchSource:0}: Error finding container 0256d300936b721b6d0dc8115e30270c93d51431e4bbce2b76b2323fedd72cdf: Status 404 returned error can't find the container with id 0256d300936b721b6d0dc8115e30270c93d51431e4bbce2b76b2323fedd72cdf Jul 24 15:53:26.154075 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.154040 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-65kzp" event={"ID":"3db27a28-ad12-4c20-b8f6-d83a6ec42a09","Type":"ContainerStarted","Data":"0256d300936b721b6d0dc8115e30270c93d51431e4bbce2b76b2323fedd72cdf"} Jul 24 15:53:26.490089 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.489996 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-799d79994b-xrnsx"] Jul 24 15:53:26.494721 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.494696 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.508039 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.508009 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-799d79994b-xrnsx"] Jul 24 15:53:26.628946 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.628914 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-console-oauth-config\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.628946 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.628948 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-console-config\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.629437 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.628967 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-oauth-serving-cert\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.629437 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.629122 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h2r6\" (UniqueName: \"kubernetes.io/projected/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-kube-api-access-5h2r6\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.629437 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.629172 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-trusted-ca-bundle\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.629437 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.629237 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-service-ca\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.629437 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.629289 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-console-serving-cert\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.730122 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.730088 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-console-oauth-config\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.730344 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.730129 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-console-config\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.730344 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.730150 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-oauth-serving-cert\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.730344 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.730257 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h2r6\" (UniqueName: \"kubernetes.io/projected/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-kube-api-access-5h2r6\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.730344 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.730295 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-trusted-ca-bundle\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.730677 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.730502 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-service-ca\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.730677 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.730576 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-console-serving-cert\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.730983 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.730922 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-console-config\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.730983 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.730928 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-oauth-serving-cert\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.731247 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.731223 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-trusted-ca-bundle\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.731324 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.731269 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-service-ca\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.732805 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.732783 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-console-oauth-config\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.732913 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.732897 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-console-serving-cert\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.739184 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.739160 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h2r6\" (UniqueName: \"kubernetes.io/projected/89fad40f-43a3-4f28-ae58-840e3d6f2d7a-kube-api-access-5h2r6\") pod \"console-799d79994b-xrnsx\" (UID: \"89fad40f-43a3-4f28-ae58-840e3d6f2d7a\") " pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:26.806034 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:26.805947 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:27.205752 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:27.205674 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-799d79994b-xrnsx"] Jul 24 15:53:27.231857 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:53:27.231827 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fad40f_43a3_4f28_ae58_840e3d6f2d7a.slice/crio-216eaff1bac96fd5f36c057752ad9b39d06ed5d6ed721e5466bc680e9544f4ab WatchSource:0}: Error finding container 216eaff1bac96fd5f36c057752ad9b39d06ed5d6ed721e5466bc680e9544f4ab: Status 404 returned error can't find the container with id 216eaff1bac96fd5f36c057752ad9b39d06ed5d6ed721e5466bc680e9544f4ab Jul 24 15:53:28.172197 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:28.172111 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-65kzp" event={"ID":"3db27a28-ad12-4c20-b8f6-d83a6ec42a09","Type":"ContainerStarted","Data":"718b7f0a80d930b9ef36b2b6841aef8806dad929d79c879066ae74cc444044a8"} Jul 24 15:53:28.172630 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:28.172198 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-65kzp" Jul 24 15:53:28.173795 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:28.173771 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799d79994b-xrnsx" event={"ID":"89fad40f-43a3-4f28-ae58-840e3d6f2d7a","Type":"ContainerStarted","Data":"9f748e1a413b8bdea167db62428c9c12f73368f679ef1e86127273f55e4af290"} Jul 24 15:53:28.173911 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:28.173809 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-799d79994b-xrnsx" event={"ID":"89fad40f-43a3-4f28-ae58-840e3d6f2d7a","Type":"ContainerStarted","Data":"216eaff1bac96fd5f36c057752ad9b39d06ed5d6ed721e5466bc680e9544f4ab"} Jul 24 15:53:28.190855 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:28.190804 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-65kzp" podStartSLOduration=1.337256817 podStartE2EDuration="3.190787088s" totalImagesPullingTime="1.853530271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:53:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:53:28.190048062 +0000 UTC m=+483.108018366" watchObservedRunningTime="2026-07-24 15:53:28.190787088 +0000 UTC m=+483.108757387" Jul 24 15:53:28.209006 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:28.208953 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-799d79994b-xrnsx" podStartSLOduration=2.208938925 podStartE2EDuration="2.208938925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:53:28.207443203 +0000 UTC m=+483.125413494" watchObservedRunningTime="2026-07-24 15:53:28.208938925 +0000 UTC m=+483.126909224" Jul 24 15:53:30.104708 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:30.104676 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7d8hj"] Jul 24 15:53:30.107178 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:30.107163 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7d8hj" Jul 24 15:53:30.110168 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:30.110146 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-gwvsn\"" Jul 24 15:53:30.120452 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:30.120423 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7d8hj"] Jul 24 15:53:30.265087 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:30.265053 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztttn\" (UniqueName: \"kubernetes.io/projected/6c8cef0e-76c4-41aa-a26d-eaa6c2963b4c-kube-api-access-ztttn\") pod \"authorino-operator-75cc8d5cb7-7d8hj\" (UID: \"6c8cef0e-76c4-41aa-a26d-eaa6c2963b4c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7d8hj" Jul 24 15:53:30.366022 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:30.365931 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztttn\" (UniqueName: \"kubernetes.io/projected/6c8cef0e-76c4-41aa-a26d-eaa6c2963b4c-kube-api-access-ztttn\") pod \"authorino-operator-75cc8d5cb7-7d8hj\" (UID: \"6c8cef0e-76c4-41aa-a26d-eaa6c2963b4c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7d8hj" Jul 24 15:53:30.375917 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:30.375892 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztttn\" (UniqueName: \"kubernetes.io/projected/6c8cef0e-76c4-41aa-a26d-eaa6c2963b4c-kube-api-access-ztttn\") pod \"authorino-operator-75cc8d5cb7-7d8hj\" (UID: \"6c8cef0e-76c4-41aa-a26d-eaa6c2963b4c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7d8hj" Jul 24 15:53:30.417573 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:30.417539 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7d8hj" Jul 24 15:53:30.564158 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:30.564128 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7d8hj"] Jul 24 15:53:30.565536 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:53:30.565501 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8cef0e_76c4_41aa_a26d_eaa6c2963b4c.slice/crio-f99f6ac1a5543c9caef40ceb861a2b2c701034b36c23cc0259dd6f7619e3d47d WatchSource:0}: Error finding container f99f6ac1a5543c9caef40ceb861a2b2c701034b36c23cc0259dd6f7619e3d47d: Status 404 returned error can't find the container with id f99f6ac1a5543c9caef40ceb861a2b2c701034b36c23cc0259dd6f7619e3d47d Jul 24 15:53:31.192181 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:31.192142 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7d8hj" event={"ID":"6c8cef0e-76c4-41aa-a26d-eaa6c2963b4c","Type":"ContainerStarted","Data":"f99f6ac1a5543c9caef40ceb861a2b2c701034b36c23cc0259dd6f7619e3d47d"} Jul 24 15:53:33.207085 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:33.207055 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7d8hj" event={"ID":"6c8cef0e-76c4-41aa-a26d-eaa6c2963b4c","Type":"ContainerStarted","Data":"13abeab7ec495688bfa4c4b0b5e0ec626f76e08cfa9a0610fc6dbccc4f0e936e"} Jul 24 15:53:33.207485 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:33.207188 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7d8hj" Jul 24 15:53:33.224980 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:33.224932 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7d8hj" podStartSLOduration=0.784894725 podStartE2EDuration="3.224915906s" totalImagesPullingTime="2.440021181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:53:33.222914425 +0000 UTC m=+488.140884724" watchObservedRunningTime="2026-07-24 15:53:33.224915906 +0000 UTC m=+488.142886206" Jul 24 15:53:34.239929 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:34.239894 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz"] Jul 24 15:53:34.242508 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:34.242491 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:53:34.246514 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:34.246492 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6l4rd\"" Jul 24 15:53:34.273862 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:34.273831 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz"] Jul 24 15:53:34.402434 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:34.402392 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9v8l\" (UniqueName: \"kubernetes.io/projected/5d84207a-7a02-4dbc-9a30-708c29a466f3-kube-api-access-n9v8l\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6m7jz\" (UID: \"5d84207a-7a02-4dbc-9a30-708c29a466f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:53:34.402629 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:34.402446 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d84207a-7a02-4dbc-9a30-708c29a466f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6m7jz\" (UID: \"5d84207a-7a02-4dbc-9a30-708c29a466f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:53:34.503070 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:34.502971 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d84207a-7a02-4dbc-9a30-708c29a466f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6m7jz\" (UID: \"5d84207a-7a02-4dbc-9a30-708c29a466f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:53:34.503273 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:34.503078 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9v8l\" (UniqueName: \"kubernetes.io/projected/5d84207a-7a02-4dbc-9a30-708c29a466f3-kube-api-access-n9v8l\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6m7jz\" (UID: \"5d84207a-7a02-4dbc-9a30-708c29a466f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:53:34.503466 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:34.503443 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d84207a-7a02-4dbc-9a30-708c29a466f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6m7jz\" (UID: \"5d84207a-7a02-4dbc-9a30-708c29a466f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:53:34.514460 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:34.514428 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9v8l\" (UniqueName: \"kubernetes.io/projected/5d84207a-7a02-4dbc-9a30-708c29a466f3-kube-api-access-n9v8l\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6m7jz\" (UID: \"5d84207a-7a02-4dbc-9a30-708c29a466f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:53:34.553009 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:34.552972 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:53:34.695026 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:34.694988 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz"] Jul 24 15:53:34.696530 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:53:34.696509 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d84207a_7a02_4dbc_9a30_708c29a466f3.slice/crio-06dfaa4df5157a7355953833ad28154b3b64516fbc317bc622def963fd55b61a WatchSource:0}: Error finding container 06dfaa4df5157a7355953833ad28154b3b64516fbc317bc622def963fd55b61a: Status 404 returned error can't find the container with id 06dfaa4df5157a7355953833ad28154b3b64516fbc317bc622def963fd55b61a Jul 24 15:53:35.221107 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:35.221072 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" event={"ID":"5d84207a-7a02-4dbc-9a30-708c29a466f3","Type":"ContainerStarted","Data":"06dfaa4df5157a7355953833ad28154b3b64516fbc317bc622def963fd55b61a"} Jul 24 15:53:36.282450 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:36.282417 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g"] Jul 24 15:53:36.285342 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:36.285323 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:53:36.288924 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:36.288905 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-54vnc\"" Jul 24 15:53:36.297244 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:36.297186 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g"] Jul 24 15:53:36.317378 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:36.317347 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmn4n\" (UniqueName: \"kubernetes.io/projected/012dd228-a720-43bb-8d72-0f5816fffcdc-kube-api-access-cmn4n\") pod \"limitador-operator-controller-manager-859d7b78bd-qzm7g\" (UID: \"012dd228-a720-43bb-8d72-0f5816fffcdc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:53:36.418571 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:36.418534 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmn4n\" (UniqueName: \"kubernetes.io/projected/012dd228-a720-43bb-8d72-0f5816fffcdc-kube-api-access-cmn4n\") pod \"limitador-operator-controller-manager-859d7b78bd-qzm7g\" (UID: \"012dd228-a720-43bb-8d72-0f5816fffcdc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:53:36.429050 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:36.429016 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmn4n\" (UniqueName: \"kubernetes.io/projected/012dd228-a720-43bb-8d72-0f5816fffcdc-kube-api-access-cmn4n\") pod \"limitador-operator-controller-manager-859d7b78bd-qzm7g\" (UID: \"012dd228-a720-43bb-8d72-0f5816fffcdc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:53:36.600198 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:36.600103 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:53:36.765276 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:36.765247 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g"] Jul 24 15:53:36.767459 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:53:36.767433 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012dd228_a720_43bb_8d72_0f5816fffcdc.slice/crio-bbdb5a5fbb24ca8e248c903afc5cabed80be2cff6e99536bf6e84782c93de114 WatchSource:0}: Error finding container bbdb5a5fbb24ca8e248c903afc5cabed80be2cff6e99536bf6e84782c93de114: Status 404 returned error can't find the container with id bbdb5a5fbb24ca8e248c903afc5cabed80be2cff6e99536bf6e84782c93de114 Jul 24 15:53:36.807157 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:36.807120 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:36.807361 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:36.807205 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:36.812025 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:36.812002 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:37.236163 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:37.236125 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" event={"ID":"012dd228-a720-43bb-8d72-0f5816fffcdc","Type":"ContainerStarted","Data":"bbdb5a5fbb24ca8e248c903afc5cabed80be2cff6e99536bf6e84782c93de114"} Jul 24 15:53:37.240737 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:37.240710 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-799d79994b-xrnsx" Jul 24 15:53:37.291802 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:37.291764 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56b965b5cd-zqgsx"] Jul 24 15:53:39.182287 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:39.182251 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-65kzp" Jul 24 15:53:40.257396 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:40.257353 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" event={"ID":"5d84207a-7a02-4dbc-9a30-708c29a466f3","Type":"ContainerStarted","Data":"7a76bcfde9df4271c44b642a32424989e00056a3bbbf1de48a96ee4d0208ab2a"} Jul 24 15:53:40.257891 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:40.257510 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:53:40.258938 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:40.258916 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" event={"ID":"012dd228-a720-43bb-8d72-0f5816fffcdc","Type":"ContainerStarted","Data":"9ce86d85a5875703f876842aebcaed4c895a1b27ae4ff5a442513575676052c0"} Jul 24 15:53:40.259056 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:40.258965 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:53:40.280194 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:40.280132 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" podStartSLOduration=1.582779156 podStartE2EDuration="6.280112623s" totalImagesPullingTime="4.697333467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:53:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:53:40.278067389 +0000 UTC m=+495.196037689" watchObservedRunningTime="2026-07-24 15:53:40.280112623 +0000 UTC m=+495.198082924" Jul 24 15:53:40.297913 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:40.297860 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" podStartSLOduration=1.682967946 podStartE2EDuration="4.297844065s" totalImagesPullingTime="2.614876119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:53:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:53:40.295768822 +0000 UTC m=+495.213739156" watchObservedRunningTime="2026-07-24 15:53:40.297844065 +0000 UTC m=+495.215814364" Jul 24 15:53:42.075079 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:42.075009 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lx98m"] Jul 24 15:53:42.077748 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:42.077728 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lx98m" Jul 24 15:53:42.080341 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:42.080314 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 24 15:53:42.080341 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:42.080330 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 24 15:53:42.080531 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:42.080367 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bd4wh\"" Jul 24 15:53:42.088947 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:42.088921 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lx98m"] Jul 24 15:53:42.163295 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:42.163258 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a29bc5d4-1424-462d-bd48-9c5f74b7c812-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lx98m\" (UID: \"a29bc5d4-1424-462d-bd48-9c5f74b7c812\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lx98m" Jul 24 15:53:42.163501 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:42.163408 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a29bc5d4-1424-462d-bd48-9c5f74b7c812-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lx98m\" (UID: \"a29bc5d4-1424-462d-bd48-9c5f74b7c812\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lx98m" Jul 24 15:53:42.163501 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:42.163453 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q27h\" (UniqueName: \"kubernetes.io/projected/a29bc5d4-1424-462d-bd48-9c5f74b7c812-kube-api-access-4q27h\") pod \"kuadrant-console-plugin-66b5774985-lx98m\" (UID: \"a29bc5d4-1424-462d-bd48-9c5f74b7c812\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lx98m" Jul 24 15:53:42.263990 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:42.263946 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a29bc5d4-1424-462d-bd48-9c5f74b7c812-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lx98m\" (UID: \"a29bc5d4-1424-462d-bd48-9c5f74b7c812\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lx98m" Jul 24 15:53:42.264197 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:42.263999 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q27h\" (UniqueName: \"kubernetes.io/projected/a29bc5d4-1424-462d-bd48-9c5f74b7c812-kube-api-access-4q27h\") pod \"kuadrant-console-plugin-66b5774985-lx98m\" (UID: \"a29bc5d4-1424-462d-bd48-9c5f74b7c812\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lx98m" Jul 24 15:53:42.264197 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:42.264034 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a29bc5d4-1424-462d-bd48-9c5f74b7c812-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lx98m\" (UID: \"a29bc5d4-1424-462d-bd48-9c5f74b7c812\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lx98m" Jul 24 15:53:42.264399 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:53:42.264199 2569 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 24 15:53:42.264399 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:53:42.264312 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a29bc5d4-1424-462d-bd48-9c5f74b7c812-plugin-serving-cert podName:a29bc5d4-1424-462d-bd48-9c5f74b7c812 nodeName:}" failed. No retries permitted until 2026-07-24 15:53:42.764288632 +0000 UTC m=+497.682258914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/a29bc5d4-1424-462d-bd48-9c5f74b7c812-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-lx98m" (UID: "a29bc5d4-1424-462d-bd48-9c5f74b7c812") : secret "plugin-serving-cert" not found Jul 24 15:53:42.264718 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:42.264690 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a29bc5d4-1424-462d-bd48-9c5f74b7c812-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lx98m\" (UID: \"a29bc5d4-1424-462d-bd48-9c5f74b7c812\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lx98m" Jul 24 15:53:42.273449 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:42.273426 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q27h\" (UniqueName: \"kubernetes.io/projected/a29bc5d4-1424-462d-bd48-9c5f74b7c812-kube-api-access-4q27h\") pod \"kuadrant-console-plugin-66b5774985-lx98m\" (UID: \"a29bc5d4-1424-462d-bd48-9c5f74b7c812\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lx98m" Jul 24 15:53:42.767018 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:42.766983 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a29bc5d4-1424-462d-bd48-9c5f74b7c812-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lx98m\" (UID: \"a29bc5d4-1424-462d-bd48-9c5f74b7c812\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lx98m" Jul 24 15:53:42.769454 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:42.769434 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a29bc5d4-1424-462d-bd48-9c5f74b7c812-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lx98m\" (UID: \"a29bc5d4-1424-462d-bd48-9c5f74b7c812\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lx98m" Jul 24 15:53:42.988680 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:42.988635 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lx98m" Jul 24 15:53:43.125359 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:43.125336 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lx98m"] Jul 24 15:53:43.127433 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:53:43.127404 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda29bc5d4_1424_462d_bd48_9c5f74b7c812.slice/crio-6b3b9c9ae6baca63efb5fa3994bec86f0cdd2534d303f7cfb2cda86c38186a70 WatchSource:0}: Error finding container 6b3b9c9ae6baca63efb5fa3994bec86f0cdd2534d303f7cfb2cda86c38186a70: Status 404 returned error can't find the container with id 6b3b9c9ae6baca63efb5fa3994bec86f0cdd2534d303f7cfb2cda86c38186a70 Jul 24 15:53:43.276640 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:43.276604 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lx98m" event={"ID":"a29bc5d4-1424-462d-bd48-9c5f74b7c812","Type":"ContainerStarted","Data":"6b3b9c9ae6baca63efb5fa3994bec86f0cdd2534d303f7cfb2cda86c38186a70"} Jul 24 15:53:44.216749 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:44.216714 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7d8hj" Jul 24 15:53:51.266330 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:51.266296 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:53:51.266768 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:51.266464 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:53:53.142183 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.142142 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz"] Jul 24 15:53:53.142632 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.142356 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" podUID="5d84207a-7a02-4dbc-9a30-708c29a466f3" containerName="manager" containerID="cri-o://7a76bcfde9df4271c44b642a32424989e00056a3bbbf1de48a96ee4d0208ab2a" gracePeriod=2 Jul 24 15:53:53.149354 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.149285 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz"] Jul 24 15:53:53.198104 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.197970 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn"] Jul 24 15:53:53.198626 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.198567 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d84207a-7a02-4dbc-9a30-708c29a466f3" containerName="manager" Jul 24 15:53:53.198626 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.198591 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d84207a-7a02-4dbc-9a30-708c29a466f3" containerName="manager" Jul 24 15:53:53.198848 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.198732 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d84207a-7a02-4dbc-9a30-708c29a466f3" containerName="manager" Jul 24 15:53:53.202520 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.202496 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn" Jul 24 15:53:53.226795 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.226767 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn"] Jul 24 15:53:53.242059 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.242033 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx"] Jul 24 15:53:53.246576 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.246031 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g"] Jul 24 15:53:53.246576 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.246244 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" podUID="012dd228-a720-43bb-8d72-0f5816fffcdc" containerName="manager" containerID="cri-o://9ce86d85a5875703f876842aebcaed4c895a1b27ae4ff5a442513575676052c0" gracePeriod=2 Jul 24 15:53:53.246576 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.246456 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx" Jul 24 15:53:53.252558 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:53:53.252518 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6m7jz\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="5d84207a-7a02-4dbc-9a30-708c29a466f3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:53:53.256983 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:53:53.256814 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6m7jz\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="5d84207a-7a02-4dbc-9a30-708c29a466f3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:53:53.262074 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:53:53.261198 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qzm7g\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="012dd228-a720-43bb-8d72-0f5816fffcdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:53:53.262074 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.261471 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g"] Jul 24 15:53:53.265105 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.265076 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx"] Jul 24 15:53:53.271361 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.271315 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/434c7588-de5a-4bfa-af19-af1cdbbf6a15-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9mkdn\" (UID: \"434c7588-de5a-4bfa-af19-af1cdbbf6a15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn" Jul 24 15:53:53.271504 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.271381 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzrjs\" (UniqueName: \"kubernetes.io/projected/434c7588-de5a-4bfa-af19-af1cdbbf6a15-kube-api-access-mzrjs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9mkdn\" (UID: \"434c7588-de5a-4bfa-af19-af1cdbbf6a15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn" Jul 24 15:53:53.286981 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.286941 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbscc"] Jul 24 15:53:53.288329 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.288304 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="012dd228-a720-43bb-8d72-0f5816fffcdc" containerName="manager" Jul 24 15:53:53.288478 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.288391 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="012dd228-a720-43bb-8d72-0f5816fffcdc" containerName="manager" Jul 24 15:53:53.288551 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.288531 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="012dd228-a720-43bb-8d72-0f5816fffcdc" containerName="manager" Jul 24 15:53:53.292065 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.292048 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbscc" Jul 24 15:53:53.294663 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:53:53.294627 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6m7jz\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="5d84207a-7a02-4dbc-9a30-708c29a466f3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:53:53.310564 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.310539 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbscc"] Jul 24 15:53:53.327155 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:53:53.327122 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qzm7g\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="012dd228-a720-43bb-8d72-0f5816fffcdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:53:53.373081 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.373037 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss7wq\" (UniqueName: \"kubernetes.io/projected/39800830-d9ce-48cb-8906-b991efdf13ff-kube-api-access-ss7wq\") pod \"limitador-operator-controller-manager-859d7b78bd-cbscc\" (UID: \"39800830-d9ce-48cb-8906-b991efdf13ff\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbscc" Jul 24 15:53:53.373261 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.373107 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68b260d5-51be-44cb-a5ab-7c2e4e275e49-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7r4rx\" (UID: \"68b260d5-51be-44cb-a5ab-7c2e4e275e49\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx" Jul 24 15:53:53.373261 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.373161 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/434c7588-de5a-4bfa-af19-af1cdbbf6a15-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9mkdn\" (UID: \"434c7588-de5a-4bfa-af19-af1cdbbf6a15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn" Jul 24 15:53:53.373261 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.373205 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmhfh\" (UniqueName: \"kubernetes.io/projected/68b260d5-51be-44cb-a5ab-7c2e4e275e49-kube-api-access-jmhfh\") pod \"kuadrant-operator-controller-manager-86878c769f-7r4rx\" (UID: \"68b260d5-51be-44cb-a5ab-7c2e4e275e49\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx" Jul 24 15:53:53.373413 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.373271 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzrjs\" (UniqueName: \"kubernetes.io/projected/434c7588-de5a-4bfa-af19-af1cdbbf6a15-kube-api-access-mzrjs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9mkdn\" (UID: \"434c7588-de5a-4bfa-af19-af1cdbbf6a15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn" Jul 24 15:53:53.373568 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.373548 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/434c7588-de5a-4bfa-af19-af1cdbbf6a15-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9mkdn\" (UID: \"434c7588-de5a-4bfa-af19-af1cdbbf6a15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn" Jul 24 15:53:53.385812 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.385783 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzrjs\" (UniqueName: \"kubernetes.io/projected/434c7588-de5a-4bfa-af19-af1cdbbf6a15-kube-api-access-mzrjs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9mkdn\" (UID: \"434c7588-de5a-4bfa-af19-af1cdbbf6a15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn" Jul 24 15:53:53.474734 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.474639 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss7wq\" (UniqueName: \"kubernetes.io/projected/39800830-d9ce-48cb-8906-b991efdf13ff-kube-api-access-ss7wq\") pod \"limitador-operator-controller-manager-859d7b78bd-cbscc\" (UID: \"39800830-d9ce-48cb-8906-b991efdf13ff\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbscc" Jul 24 15:53:53.474734 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.474699 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68b260d5-51be-44cb-a5ab-7c2e4e275e49-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7r4rx\" (UID: \"68b260d5-51be-44cb-a5ab-7c2e4e275e49\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx" Jul 24 15:53:53.474963 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.474758 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmhfh\" (UniqueName: \"kubernetes.io/projected/68b260d5-51be-44cb-a5ab-7c2e4e275e49-kube-api-access-jmhfh\") pod \"kuadrant-operator-controller-manager-86878c769f-7r4rx\" (UID: \"68b260d5-51be-44cb-a5ab-7c2e4e275e49\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx" Jul 24 15:53:53.475142 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.475108 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68b260d5-51be-44cb-a5ab-7c2e4e275e49-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7r4rx\" (UID: \"68b260d5-51be-44cb-a5ab-7c2e4e275e49\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx" Jul 24 15:53:53.488025 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.487997 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmhfh\" (UniqueName: \"kubernetes.io/projected/68b260d5-51be-44cb-a5ab-7c2e4e275e49-kube-api-access-jmhfh\") pod \"kuadrant-operator-controller-manager-86878c769f-7r4rx\" (UID: \"68b260d5-51be-44cb-a5ab-7c2e4e275e49\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx" Jul 24 15:53:53.488242 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.488157 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss7wq\" (UniqueName: \"kubernetes.io/projected/39800830-d9ce-48cb-8906-b991efdf13ff-kube-api-access-ss7wq\") pod \"limitador-operator-controller-manager-859d7b78bd-cbscc\" (UID: \"39800830-d9ce-48cb-8906-b991efdf13ff\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbscc" Jul 24 15:53:53.541194 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.541162 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn" Jul 24 15:53:53.563160 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.563121 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx" Jul 24 15:53:53.622080 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:53:53.622047 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbscc" Jul 24 15:54:02.320304 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:02.320250 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56b965b5cd-zqgsx" podUID="6b348b56-c220-40b8-907b-da9b58719243" containerName="console" containerID="cri-o://fc5a9ba969ff38a3a2f40ccc0f91c46c5b0d6190f21aa05279e8a11836363675" gracePeriod=15 Jul 24 15:54:04.154265 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.153879 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:54:04.161247 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:04.158568 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qzm7g\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="012dd228-a720-43bb-8d72-0f5816fffcdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:54:04.203257 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.203228 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:54:04.205935 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:04.205771 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6m7jz\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="5d84207a-7a02-4dbc-9a30-708c29a466f3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:54:04.208341 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:04.208277 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qzm7g\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="012dd228-a720-43bb-8d72-0f5816fffcdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:54:04.237150 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.236618 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:54:04.239285 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:04.239186 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6m7jz\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="5d84207a-7a02-4dbc-9a30-708c29a466f3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:54:04.265011 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:04.264964 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qzm7g\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="012dd228-a720-43bb-8d72-0f5816fffcdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:54:04.287877 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.287837 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d84207a-7a02-4dbc-9a30-708c29a466f3-extensions-socket-volume\") pod \"5d84207a-7a02-4dbc-9a30-708c29a466f3\" (UID: \"5d84207a-7a02-4dbc-9a30-708c29a466f3\") " Jul 24 15:54:04.288060 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.287887 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9v8l\" (UniqueName: \"kubernetes.io/projected/5d84207a-7a02-4dbc-9a30-708c29a466f3-kube-api-access-n9v8l\") pod \"5d84207a-7a02-4dbc-9a30-708c29a466f3\" (UID: \"5d84207a-7a02-4dbc-9a30-708c29a466f3\") " Jul 24 15:54:04.288060 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.287926 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmn4n\" (UniqueName: \"kubernetes.io/projected/012dd228-a720-43bb-8d72-0f5816fffcdc-kube-api-access-cmn4n\") pod \"012dd228-a720-43bb-8d72-0f5816fffcdc\" (UID: \"012dd228-a720-43bb-8d72-0f5816fffcdc\") " Jul 24 15:54:04.288520 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.288470 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d84207a-7a02-4dbc-9a30-708c29a466f3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5d84207a-7a02-4dbc-9a30-708c29a466f3" (UID: "5d84207a-7a02-4dbc-9a30-708c29a466f3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:54:04.291364 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.291340 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d84207a-7a02-4dbc-9a30-708c29a466f3-kube-api-access-n9v8l" (OuterVolumeSpecName: "kube-api-access-n9v8l") pod "5d84207a-7a02-4dbc-9a30-708c29a466f3" (UID: "5d84207a-7a02-4dbc-9a30-708c29a466f3"). InnerVolumeSpecName "kube-api-access-n9v8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:54:04.293232 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.293193 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012dd228-a720-43bb-8d72-0f5816fffcdc-kube-api-access-cmn4n" (OuterVolumeSpecName: "kube-api-access-cmn4n") pod "012dd228-a720-43bb-8d72-0f5816fffcdc" (UID: "012dd228-a720-43bb-8d72-0f5816fffcdc"). InnerVolumeSpecName "kube-api-access-cmn4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:54:04.307031 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.307000 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx"] Jul 24 15:54:04.309187 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:54:04.309158 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b260d5_51be_44cb_a5ab_7c2e4e275e49.slice/crio-e0e8ab3c716fe6d175a224eff1d8817c464d5bb1ccf1ac863145b78ad688e869 WatchSource:0}: Error finding container e0e8ab3c716fe6d175a224eff1d8817c464d5bb1ccf1ac863145b78ad688e869: Status 404 returned error can't find the container with id e0e8ab3c716fe6d175a224eff1d8817c464d5bb1ccf1ac863145b78ad688e869 Jul 24 15:54:04.388576 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.388551 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b348b56-c220-40b8-907b-da9b58719243-console-oauth-config\") pod \"6b348b56-c220-40b8-907b-da9b58719243\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " Jul 24 15:54:04.388686 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.388611 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b348b56-c220-40b8-907b-da9b58719243-console-serving-cert\") pod \"6b348b56-c220-40b8-907b-da9b58719243\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " Jul 24 15:54:04.388686 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.388649 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-console-config\") pod \"6b348b56-c220-40b8-907b-da9b58719243\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " Jul 24 15:54:04.388686 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.388683 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-service-ca\") pod \"6b348b56-c220-40b8-907b-da9b58719243\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " Jul 24 15:54:04.388805 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.388765 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-oauth-serving-cert\") pod \"6b348b56-c220-40b8-907b-da9b58719243\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " Jul 24 15:54:04.388854 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.388810 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv76n\" (UniqueName: \"kubernetes.io/projected/6b348b56-c220-40b8-907b-da9b58719243-kube-api-access-mv76n\") pod \"6b348b56-c220-40b8-907b-da9b58719243\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " Jul 24 15:54:04.388854 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.388831 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-trusted-ca-bundle\") pod \"6b348b56-c220-40b8-907b-da9b58719243\" (UID: \"6b348b56-c220-40b8-907b-da9b58719243\") " Jul 24 15:54:04.389186 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.389080 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-service-ca" (OuterVolumeSpecName: "service-ca") pod "6b348b56-c220-40b8-907b-da9b58719243" (UID: "6b348b56-c220-40b8-907b-da9b58719243"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:54:04.389186 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.389084 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-console-config" (OuterVolumeSpecName: "console-config") pod "6b348b56-c220-40b8-907b-da9b58719243" (UID: "6b348b56-c220-40b8-907b-da9b58719243"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:54:04.389186 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.389140 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6b348b56-c220-40b8-907b-da9b58719243" (UID: "6b348b56-c220-40b8-907b-da9b58719243"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:54:04.389186 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.389158 2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d84207a-7a02-4dbc-9a30-708c29a466f3-extensions-socket-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:54:04.389186 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.389177 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9v8l\" (UniqueName: \"kubernetes.io/projected/5d84207a-7a02-4dbc-9a30-708c29a466f3-kube-api-access-n9v8l\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:54:04.389549 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.389204 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmn4n\" (UniqueName: \"kubernetes.io/projected/012dd228-a720-43bb-8d72-0f5816fffcdc-kube-api-access-cmn4n\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:54:04.389549 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.389401 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6b348b56-c220-40b8-907b-da9b58719243" (UID: "6b348b56-c220-40b8-907b-da9b58719243"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:54:04.390621 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.390592 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b348b56-c220-40b8-907b-da9b58719243-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6b348b56-c220-40b8-907b-da9b58719243" (UID: "6b348b56-c220-40b8-907b-da9b58719243"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:54:04.390735 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.390697 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b348b56-c220-40b8-907b-da9b58719243-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6b348b56-c220-40b8-907b-da9b58719243" (UID: "6b348b56-c220-40b8-907b-da9b58719243"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:54:04.390885 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.390865 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b348b56-c220-40b8-907b-da9b58719243-kube-api-access-mv76n" (OuterVolumeSpecName: "kube-api-access-mv76n") pod "6b348b56-c220-40b8-907b-da9b58719243" (UID: "6b348b56-c220-40b8-907b-da9b58719243"). InnerVolumeSpecName "kube-api-access-mv76n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:54:04.422759 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.422727 2569 generic.go:356] "Generic (PLEG): container finished" podID="5d84207a-7a02-4dbc-9a30-708c29a466f3" containerID="7a76bcfde9df4271c44b642a32424989e00056a3bbbf1de48a96ee4d0208ab2a" exitCode=0 Jul 24 15:54:04.422890 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.422772 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:54:04.422890 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.422827 2569 scope.go:117] "RemoveContainer" containerID="7a76bcfde9df4271c44b642a32424989e00056a3bbbf1de48a96ee4d0208ab2a" Jul 24 15:54:04.424818 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.424793 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx" event={"ID":"68b260d5-51be-44cb-a5ab-7c2e4e275e49","Type":"ContainerStarted","Data":"1d5b8100aaf1116cc5e799aa39dcd5c365d8b0d23385ebc9c2fca547d655de93"} Jul 24 15:54:04.424941 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.424837 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx" event={"ID":"68b260d5-51be-44cb-a5ab-7c2e4e275e49","Type":"ContainerStarted","Data":"e0e8ab3c716fe6d175a224eff1d8817c464d5bb1ccf1ac863145b78ad688e869"} Jul 24 15:54:04.424941 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.424859 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx" Jul 24 15:54:04.425377 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:04.425346 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6m7jz\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="5d84207a-7a02-4dbc-9a30-708c29a466f3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:54:04.428528 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:04.427855 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qzm7g\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="012dd228-a720-43bb-8d72-0f5816fffcdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:54:04.428528 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.428226 2569 generic.go:356] "Generic (PLEG): container finished" podID="6b348b56-c220-40b8-907b-da9b58719243" containerID="fc5a9ba969ff38a3a2f40ccc0f91c46c5b0d6190f21aa05279e8a11836363675" exitCode=0 Jul 24 15:54:04.428528 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.428330 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56b965b5cd-zqgsx" Jul 24 15:54:04.428528 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.428396 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b965b5cd-zqgsx" event={"ID":"6b348b56-c220-40b8-907b-da9b58719243","Type":"ContainerDied","Data":"fc5a9ba969ff38a3a2f40ccc0f91c46c5b0d6190f21aa05279e8a11836363675"} Jul 24 15:54:04.428528 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.428457 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56b965b5cd-zqgsx" event={"ID":"6b348b56-c220-40b8-907b-da9b58719243","Type":"ContainerDied","Data":"f61d3c5300456c7dbf740ea5750cf0ea844ebe449c90455d66579cf1ab1c7b66"} Jul 24 15:54:04.431741 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:04.431617 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6m7jz\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="5d84207a-7a02-4dbc-9a30-708c29a466f3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:54:04.433246 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.433043 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lx98m" event={"ID":"a29bc5d4-1424-462d-bd48-9c5f74b7c812","Type":"ContainerStarted","Data":"1807a592c053d7296e0dca8ad79d5948afc5ad761405c6d3c776e96de6d08e7a"} Jul 24 15:54:04.433729 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:04.433700 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qzm7g\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="012dd228-a720-43bb-8d72-0f5816fffcdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:54:04.434914 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.434894 2569 generic.go:356] "Generic (PLEG): container finished" podID="012dd228-a720-43bb-8d72-0f5816fffcdc" containerID="9ce86d85a5875703f876842aebcaed4c895a1b27ae4ff5a442513575676052c0" exitCode=0 Jul 24 15:54:04.435009 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.434941 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:54:04.439164 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.439146 2569 scope.go:117] "RemoveContainer" containerID="7a76bcfde9df4271c44b642a32424989e00056a3bbbf1de48a96ee4d0208ab2a" Jul 24 15:54:04.439533 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:04.439508 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a76bcfde9df4271c44b642a32424989e00056a3bbbf1de48a96ee4d0208ab2a\": container with ID starting with 7a76bcfde9df4271c44b642a32424989e00056a3bbbf1de48a96ee4d0208ab2a not found: ID does not exist" containerID="7a76bcfde9df4271c44b642a32424989e00056a3bbbf1de48a96ee4d0208ab2a" Jul 24 15:54:04.439630 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.439543 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a76bcfde9df4271c44b642a32424989e00056a3bbbf1de48a96ee4d0208ab2a"} err="failed to get container status \"7a76bcfde9df4271c44b642a32424989e00056a3bbbf1de48a96ee4d0208ab2a\": rpc error: code = NotFound desc = could not find container \"7a76bcfde9df4271c44b642a32424989e00056a3bbbf1de48a96ee4d0208ab2a\": container with ID starting with 7a76bcfde9df4271c44b642a32424989e00056a3bbbf1de48a96ee4d0208ab2a not found: ID does not exist" Jul 24 15:54:04.439630 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.439566 2569 scope.go:117] "RemoveContainer" containerID="fc5a9ba969ff38a3a2f40ccc0f91c46c5b0d6190f21aa05279e8a11836363675" Jul 24 15:54:04.449802 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.449779 2569 scope.go:117] "RemoveContainer" containerID="fc5a9ba969ff38a3a2f40ccc0f91c46c5b0d6190f21aa05279e8a11836363675" Jul 24 15:54:04.450074 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:04.450056 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc5a9ba969ff38a3a2f40ccc0f91c46c5b0d6190f21aa05279e8a11836363675\": container with ID starting with fc5a9ba969ff38a3a2f40ccc0f91c46c5b0d6190f21aa05279e8a11836363675 not found: ID does not exist" containerID="fc5a9ba969ff38a3a2f40ccc0f91c46c5b0d6190f21aa05279e8a11836363675" Jul 24 15:54:04.450125 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.450082 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc5a9ba969ff38a3a2f40ccc0f91c46c5b0d6190f21aa05279e8a11836363675"} err="failed to get container status \"fc5a9ba969ff38a3a2f40ccc0f91c46c5b0d6190f21aa05279e8a11836363675\": rpc error: code = NotFound desc = could not find container \"fc5a9ba969ff38a3a2f40ccc0f91c46c5b0d6190f21aa05279e8a11836363675\": container with ID starting with fc5a9ba969ff38a3a2f40ccc0f91c46c5b0d6190f21aa05279e8a11836363675 not found: ID does not exist" Jul 24 15:54:04.450125 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.450098 2569 scope.go:117] "RemoveContainer" containerID="9ce86d85a5875703f876842aebcaed4c895a1b27ae4ff5a442513575676052c0" Jul 24 15:54:04.458293 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:04.458265 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6m7jz\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="5d84207a-7a02-4dbc-9a30-708c29a466f3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:54:04.458544 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.458501 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx" podStartSLOduration=11.458483669 podStartE2EDuration="11.458483669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:54:04.455932187 +0000 UTC m=+519.373902486" watchObservedRunningTime="2026-07-24 15:54:04.458483669 +0000 UTC m=+519.376453969" Jul 24 15:54:04.460013 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.459997 2569 scope.go:117] "RemoveContainer" containerID="9ce86d85a5875703f876842aebcaed4c895a1b27ae4ff5a442513575676052c0" Jul 24 15:54:04.460290 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:04.460256 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ce86d85a5875703f876842aebcaed4c895a1b27ae4ff5a442513575676052c0\": container with ID starting with 9ce86d85a5875703f876842aebcaed4c895a1b27ae4ff5a442513575676052c0 not found: ID does not exist" containerID="9ce86d85a5875703f876842aebcaed4c895a1b27ae4ff5a442513575676052c0" Jul 24 15:54:04.460342 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.460302 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce86d85a5875703f876842aebcaed4c895a1b27ae4ff5a442513575676052c0"} err="failed to get container status \"9ce86d85a5875703f876842aebcaed4c895a1b27ae4ff5a442513575676052c0\": rpc error: code = NotFound desc = could not find container \"9ce86d85a5875703f876842aebcaed4c895a1b27ae4ff5a442513575676052c0\": container with ID starting with 9ce86d85a5875703f876842aebcaed4c895a1b27ae4ff5a442513575676052c0 not found: ID does not exist" Jul 24 15:54:04.477431 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.477410 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56b965b5cd-zqgsx"] Jul 24 15:54:04.480812 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:04.480788 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qzm7g\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="012dd228-a720-43bb-8d72-0f5816fffcdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:54:04.482049 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.482030 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56b965b5cd-zqgsx"] Jul 24 15:54:04.490382 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.490362 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b348b56-c220-40b8-907b-da9b58719243-console-serving-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:54:04.490470 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.490384 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-console-config\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:54:04.490470 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.490399 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-service-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:54:04.490470 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.490411 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-oauth-serving-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:54:04.490470 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.490422 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv76n\" (UniqueName: \"kubernetes.io/projected/6b348b56-c220-40b8-907b-da9b58719243-kube-api-access-mv76n\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:54:04.490470 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.490435 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b348b56-c220-40b8-907b-da9b58719243-trusted-ca-bundle\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:54:04.490470 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.490448 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b348b56-c220-40b8-907b-da9b58719243-console-oauth-config\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:54:04.538672 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.538627 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lx98m" podStartSLOduration=1.54026636 podStartE2EDuration="22.538611583s" totalImagesPullingTime="20.998345223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:54:04.498360611 +0000 UTC m=+519.416330910" watchObservedRunningTime="2026-07-24 15:54:04.538611583 +0000 UTC m=+519.456581883" Jul 24 15:54:04.540429 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.540391 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbscc"] Jul 24 15:54:04.542125 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:54:04.542097 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39800830_d9ce_48cb_8906_b991efdf13ff.slice/crio-43cc851177a63ebfac5466ead4066eac34670dc63030b9afb5b2090bf5da0832 WatchSource:0}: Error finding container 43cc851177a63ebfac5466ead4066eac34670dc63030b9afb5b2090bf5da0832: Status 404 returned error can't find the container with id 43cc851177a63ebfac5466ead4066eac34670dc63030b9afb5b2090bf5da0832 Jul 24 15:54:04.544306 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:04.544286 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn"] Jul 24 15:54:04.545071 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:54:04.545048 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434c7588_de5a_4bfa_af19_af1cdbbf6a15.slice/crio-2d40eabe3787bcced09420bcdd97d1fa05254d2ba2d5bcfc513f695b09b95c42 WatchSource:0}: Error finding container 2d40eabe3787bcced09420bcdd97d1fa05254d2ba2d5bcfc513f695b09b95c42: Status 404 returned error can't find the container with id 2d40eabe3787bcced09420bcdd97d1fa05254d2ba2d5bcfc513f695b09b95c42 Jul 24 15:54:05.444119 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:05.444080 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn" event={"ID":"434c7588-de5a-4bfa-af19-af1cdbbf6a15","Type":"ContainerStarted","Data":"112c806019b01ae44b44c4a6b145d6c20b4d71ca5c33d14c55e6b9c7bec5d581"} Jul 24 15:54:05.444591 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:05.444145 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn" event={"ID":"434c7588-de5a-4bfa-af19-af1cdbbf6a15","Type":"ContainerStarted","Data":"2d40eabe3787bcced09420bcdd97d1fa05254d2ba2d5bcfc513f695b09b95c42"} Jul 24 15:54:05.444591 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:05.444189 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn" Jul 24 15:54:05.445630 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:05.445599 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbscc" event={"ID":"39800830-d9ce-48cb-8906-b991efdf13ff","Type":"ContainerStarted","Data":"7bb46dbe3c83c0613bc902bff005b47e0c282f97b6ad29f81af8104bca173351"} Jul 24 15:54:05.445790 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:05.445657 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbscc" event={"ID":"39800830-d9ce-48cb-8906-b991efdf13ff","Type":"ContainerStarted","Data":"43cc851177a63ebfac5466ead4066eac34670dc63030b9afb5b2090bf5da0832"} Jul 24 15:54:05.445790 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:05.445687 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbscc" Jul 24 15:54:05.464512 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:05.464460 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn" podStartSLOduration=12.464444861 podStartE2EDuration="12.464444861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:54:05.464059955 +0000 UTC m=+520.382030267" watchObservedRunningTime="2026-07-24 15:54:05.464444861 +0000 UTC m=+520.382415160" Jul 24 15:54:05.466012 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:05.465983 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6m7jz\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="5d84207a-7a02-4dbc-9a30-708c29a466f3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:54:05.467772 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:05.467751 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qzm7g\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="012dd228-a720-43bb-8d72-0f5816fffcdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:54:05.486533 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:05.486483 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbscc" podStartSLOduration=12.486468572 podStartE2EDuration="12.486468572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:54:05.484636948 +0000 UTC m=+520.402607248" watchObservedRunningTime="2026-07-24 15:54:05.486468572 +0000 UTC m=+520.404438953" Jul 24 15:54:05.486731 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:05.486545 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qzm7g\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="012dd228-a720-43bb-8d72-0f5816fffcdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:54:05.488412 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:05.488390 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6m7jz\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="5d84207a-7a02-4dbc-9a30-708c29a466f3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:54:05.626831 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:05.626789 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6m7jz\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="5d84207a-7a02-4dbc-9a30-708c29a466f3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6m7jz" Jul 24 15:54:05.628462 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:05.628433 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="012dd228-a720-43bb-8d72-0f5816fffcdc" path="/var/lib/kubelet/pods/012dd228-a720-43bb-8d72-0f5816fffcdc/volumes" Jul 24 15:54:05.628857 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:05.628842 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d84207a-7a02-4dbc-9a30-708c29a466f3" path="/var/lib/kubelet/pods/5d84207a-7a02-4dbc-9a30-708c29a466f3/volumes" Jul 24 15:54:05.629265 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:05.629225 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-qzm7g\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="012dd228-a720-43bb-8d72-0f5816fffcdc" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-qzm7g" Jul 24 15:54:05.629401 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:05.629308 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b348b56-c220-40b8-907b-da9b58719243" path="/var/lib/kubelet/pods/6b348b56-c220-40b8-907b-da9b58719243/volumes" Jul 24 15:54:15.450594 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:15.450557 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx" Jul 24 15:54:15.503291 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:15.503259 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn"] Jul 24 15:54:15.503554 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:15.503520 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn" podUID="434c7588-de5a-4bfa-af19-af1cdbbf6a15" containerName="manager" containerID="cri-o://112c806019b01ae44b44c4a6b145d6c20b4d71ca5c33d14c55e6b9c7bec5d581" gracePeriod=10 Jul 24 15:54:15.505129 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:15.505108 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn" Jul 24 15:54:15.762990 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:15.762968 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn" Jul 24 15:54:15.893079 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:15.893032 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/434c7588-de5a-4bfa-af19-af1cdbbf6a15-extensions-socket-volume\") pod \"434c7588-de5a-4bfa-af19-af1cdbbf6a15\" (UID: \"434c7588-de5a-4bfa-af19-af1cdbbf6a15\") " Jul 24 15:54:15.893079 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:15.893088 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzrjs\" (UniqueName: \"kubernetes.io/projected/434c7588-de5a-4bfa-af19-af1cdbbf6a15-kube-api-access-mzrjs\") pod \"434c7588-de5a-4bfa-af19-af1cdbbf6a15\" (UID: \"434c7588-de5a-4bfa-af19-af1cdbbf6a15\") " Jul 24 15:54:15.893552 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:15.893526 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434c7588-de5a-4bfa-af19-af1cdbbf6a15-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "434c7588-de5a-4bfa-af19-af1cdbbf6a15" (UID: "434c7588-de5a-4bfa-af19-af1cdbbf6a15"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:54:15.895262 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:15.895228 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434c7588-de5a-4bfa-af19-af1cdbbf6a15-kube-api-access-mzrjs" (OuterVolumeSpecName: "kube-api-access-mzrjs") pod "434c7588-de5a-4bfa-af19-af1cdbbf6a15" (UID: "434c7588-de5a-4bfa-af19-af1cdbbf6a15"). InnerVolumeSpecName "kube-api-access-mzrjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:54:15.993895 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:15.993809 2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/434c7588-de5a-4bfa-af19-af1cdbbf6a15-extensions-socket-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:54:15.993895 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:15.993835 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzrjs\" (UniqueName: \"kubernetes.io/projected/434c7588-de5a-4bfa-af19-af1cdbbf6a15-kube-api-access-mzrjs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:54:16.456049 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:16.456018 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cbscc" Jul 24 15:54:16.525838 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:16.525806 2569 generic.go:356] "Generic (PLEG): container finished" podID="434c7588-de5a-4bfa-af19-af1cdbbf6a15" containerID="112c806019b01ae44b44c4a6b145d6c20b4d71ca5c33d14c55e6b9c7bec5d581" exitCode=0 Jul 24 15:54:16.526033 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:16.525863 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn" Jul 24 15:54:16.526033 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:16.525882 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn" event={"ID":"434c7588-de5a-4bfa-af19-af1cdbbf6a15","Type":"ContainerDied","Data":"112c806019b01ae44b44c4a6b145d6c20b4d71ca5c33d14c55e6b9c7bec5d581"} Jul 24 15:54:16.526033 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:16.525971 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn" event={"ID":"434c7588-de5a-4bfa-af19-af1cdbbf6a15","Type":"ContainerDied","Data":"2d40eabe3787bcced09420bcdd97d1fa05254d2ba2d5bcfc513f695b09b95c42"} Jul 24 15:54:16.526033 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:16.525999 2569 scope.go:117] "RemoveContainer" containerID="112c806019b01ae44b44c4a6b145d6c20b4d71ca5c33d14c55e6b9c7bec5d581" Jul 24 15:54:16.536489 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:16.536466 2569 scope.go:117] "RemoveContainer" containerID="112c806019b01ae44b44c4a6b145d6c20b4d71ca5c33d14c55e6b9c7bec5d581" Jul 24 15:54:16.536743 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:16.536723 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"112c806019b01ae44b44c4a6b145d6c20b4d71ca5c33d14c55e6b9c7bec5d581\": container with ID starting with 112c806019b01ae44b44c4a6b145d6c20b4d71ca5c33d14c55e6b9c7bec5d581 not found: ID does not exist" containerID="112c806019b01ae44b44c4a6b145d6c20b4d71ca5c33d14c55e6b9c7bec5d581" Jul 24 15:54:16.536812 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:16.536755 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"112c806019b01ae44b44c4a6b145d6c20b4d71ca5c33d14c55e6b9c7bec5d581"} err="failed to get container status \"112c806019b01ae44b44c4a6b145d6c20b4d71ca5c33d14c55e6b9c7bec5d581\": rpc error: code = NotFound desc = could not find container \"112c806019b01ae44b44c4a6b145d6c20b4d71ca5c33d14c55e6b9c7bec5d581\": container with ID starting with 112c806019b01ae44b44c4a6b145d6c20b4d71ca5c33d14c55e6b9c7bec5d581 not found: ID does not exist" Jul 24 15:54:16.550980 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:16.550953 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn"] Jul 24 15:54:16.557893 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:16.557868 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9mkdn"] Jul 24 15:54:17.628035 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:17.628002 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="434c7588-de5a-4bfa-af19-af1cdbbf6a15" path="/var/lib/kubelet/pods/434c7588-de5a-4bfa-af19-af1cdbbf6a15/volumes" Jul 24 15:54:33.539814 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:33.539775 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ggdsf"] Jul 24 15:54:33.540367 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:33.540320 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b348b56-c220-40b8-907b-da9b58719243" containerName="console" Jul 24 15:54:33.540367 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:33.540335 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b348b56-c220-40b8-907b-da9b58719243" containerName="console" Jul 24 15:54:33.540367 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:33.540345 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="434c7588-de5a-4bfa-af19-af1cdbbf6a15" containerName="manager" Jul 24 15:54:33.540367 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:33.540350 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="434c7588-de5a-4bfa-af19-af1cdbbf6a15" containerName="manager" Jul 24 15:54:33.540527 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:33.540466 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="434c7588-de5a-4bfa-af19-af1cdbbf6a15" containerName="manager" Jul 24 15:54:33.540527 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:33.540476 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b348b56-c220-40b8-907b-da9b58719243" containerName="console" Jul 24 15:54:33.657924 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:33.657888 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ggdsf" Jul 24 15:54:33.660344 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:33.660320 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 24 15:54:33.665506 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:33.665483 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ggdsf"] Jul 24 15:54:33.665649 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:33.665526 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ggdsf"] Jul 24 15:54:33.750452 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:33.750415 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed6ff5c0-d89f-4154-8ff3-e365551e12da-config-file\") pod \"limitador-limitador-64c94fd64f-ggdsf\" (UID: \"ed6ff5c0-d89f-4154-8ff3-e365551e12da\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ggdsf" Jul 24 15:54:33.750650 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:33.750476 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8ntd\" (UniqueName: \"kubernetes.io/projected/ed6ff5c0-d89f-4154-8ff3-e365551e12da-kube-api-access-c8ntd\") pod \"limitador-limitador-64c94fd64f-ggdsf\" (UID: \"ed6ff5c0-d89f-4154-8ff3-e365551e12da\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ggdsf" Jul 24 15:54:33.851236 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:33.851125 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed6ff5c0-d89f-4154-8ff3-e365551e12da-config-file\") pod \"limitador-limitador-64c94fd64f-ggdsf\" (UID: \"ed6ff5c0-d89f-4154-8ff3-e365551e12da\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ggdsf" Jul 24 15:54:33.851236 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:33.851171 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8ntd\" (UniqueName: \"kubernetes.io/projected/ed6ff5c0-d89f-4154-8ff3-e365551e12da-kube-api-access-c8ntd\") pod \"limitador-limitador-64c94fd64f-ggdsf\" (UID: \"ed6ff5c0-d89f-4154-8ff3-e365551e12da\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ggdsf" Jul 24 15:54:33.851770 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:33.851747 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed6ff5c0-d89f-4154-8ff3-e365551e12da-config-file\") pod \"limitador-limitador-64c94fd64f-ggdsf\" (UID: \"ed6ff5c0-d89f-4154-8ff3-e365551e12da\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ggdsf" Jul 24 15:54:33.859449 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:33.859419 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8ntd\" (UniqueName: \"kubernetes.io/projected/ed6ff5c0-d89f-4154-8ff3-e365551e12da-kube-api-access-c8ntd\") pod \"limitador-limitador-64c94fd64f-ggdsf\" (UID: \"ed6ff5c0-d89f-4154-8ff3-e365551e12da\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ggdsf" Jul 24 15:54:33.971353 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:33.971310 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ggdsf" Jul 24 15:54:34.111878 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:34.111844 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ggdsf"] Jul 24 15:54:34.113701 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:54:34.113676 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6ff5c0_d89f_4154_8ff3_e365551e12da.slice/crio-71145b6c8773c2d18e60e2528e49c34ddf63d43fa000664a7290010160105689 WatchSource:0}: Error finding container 71145b6c8773c2d18e60e2528e49c34ddf63d43fa000664a7290010160105689: Status 404 returned error can't find the container with id 71145b6c8773c2d18e60e2528e49c34ddf63d43fa000664a7290010160105689 Jul 24 15:54:34.350401 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:34.350369 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-96vtg"] Jul 24 15:54:34.353601 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:34.353580 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-96vtg" Jul 24 15:54:34.356097 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:34.356068 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wqb4g\"" Jul 24 15:54:34.363959 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:34.363940 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-96vtg"] Jul 24 15:54:34.455838 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:34.455804 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4fwx\" (UniqueName: \"kubernetes.io/projected/6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a-kube-api-access-p4fwx\") pod \"authorino-58cd955675-96vtg\" (UID: \"6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a\") " pod="kuadrant-system/authorino-58cd955675-96vtg" Jul 24 15:54:34.556873 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:34.556838 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4fwx\" (UniqueName: \"kubernetes.io/projected/6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a-kube-api-access-p4fwx\") pod \"authorino-58cd955675-96vtg\" (UID: \"6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a\") " pod="kuadrant-system/authorino-58cd955675-96vtg" Jul 24 15:54:34.561258 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:34.561188 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-4bw9l"] Jul 24 15:54:34.564036 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:34.564010 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4bw9l" Jul 24 15:54:34.566088 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:34.566070 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4fwx\" (UniqueName: \"kubernetes.io/projected/6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a-kube-api-access-p4fwx\") pod \"authorino-58cd955675-96vtg\" (UID: \"6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a\") " pod="kuadrant-system/authorino-58cd955675-96vtg" Jul 24 15:54:34.573680 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:34.573656 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4bw9l"] Jul 24 15:54:34.640504 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:34.640476 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ggdsf" event={"ID":"ed6ff5c0-d89f-4154-8ff3-e365551e12da","Type":"ContainerStarted","Data":"71145b6c8773c2d18e60e2528e49c34ddf63d43fa000664a7290010160105689"} Jul 24 15:54:34.658151 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:34.658118 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4n7w\" (UniqueName: \"kubernetes.io/projected/ab904abe-80d4-4e21-a944-383fb364f361-kube-api-access-j4n7w\") pod \"authorino-75d556bd77-4bw9l\" (UID: \"ab904abe-80d4-4e21-a944-383fb364f361\") " pod="kuadrant-system/authorino-75d556bd77-4bw9l" Jul 24 15:54:34.663935 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:34.663911 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-96vtg" Jul 24 15:54:34.759413 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:34.759255 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4n7w\" (UniqueName: \"kubernetes.io/projected/ab904abe-80d4-4e21-a944-383fb364f361-kube-api-access-j4n7w\") pod \"authorino-75d556bd77-4bw9l\" (UID: \"ab904abe-80d4-4e21-a944-383fb364f361\") " pod="kuadrant-system/authorino-75d556bd77-4bw9l" Jul 24 15:54:34.768301 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:34.768235 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4n7w\" (UniqueName: \"kubernetes.io/projected/ab904abe-80d4-4e21-a944-383fb364f361-kube-api-access-j4n7w\") pod \"authorino-75d556bd77-4bw9l\" (UID: \"ab904abe-80d4-4e21-a944-383fb364f361\") " pod="kuadrant-system/authorino-75d556bd77-4bw9l" Jul 24 15:54:34.832600 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:34.832558 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-96vtg"] Jul 24 15:54:34.834740 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:54:34.834710 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a8bf3fc_fb2d_4606_9ee4_3598ee63d45a.slice/crio-b038f0010f402e86c865e64c92e07bfed2494ada285329e7a1a5908d9e922112 WatchSource:0}: Error finding container b038f0010f402e86c865e64c92e07bfed2494ada285329e7a1a5908d9e922112: Status 404 returned error can't find the container with id b038f0010f402e86c865e64c92e07bfed2494ada285329e7a1a5908d9e922112 Jul 24 15:54:34.884375 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:34.884342 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4bw9l" Jul 24 15:54:35.077831 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:35.077700 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4bw9l"] Jul 24 15:54:35.081142 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:54:35.081114 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab904abe_80d4_4e21_a944_383fb364f361.slice/crio-049d2a91a9e96759b1bc6822c3871b82eeaf7c0b64452ba80e8e76d0a04751ac WatchSource:0}: Error finding container 049d2a91a9e96759b1bc6822c3871b82eeaf7c0b64452ba80e8e76d0a04751ac: Status 404 returned error can't find the container with id 049d2a91a9e96759b1bc6822c3871b82eeaf7c0b64452ba80e8e76d0a04751ac Jul 24 15:54:35.656165 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:35.656129 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-96vtg" event={"ID":"6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a","Type":"ContainerStarted","Data":"b038f0010f402e86c865e64c92e07bfed2494ada285329e7a1a5908d9e922112"} Jul 24 15:54:35.659303 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:35.659202 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4bw9l" event={"ID":"ab904abe-80d4-4e21-a944-383fb364f361","Type":"ContainerStarted","Data":"049d2a91a9e96759b1bc6822c3871b82eeaf7c0b64452ba80e8e76d0a04751ac"} Jul 24 15:54:39.687099 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:39.687057 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4bw9l" event={"ID":"ab904abe-80d4-4e21-a944-383fb364f361","Type":"ContainerStarted","Data":"3657db68e3a45e8b9c58f9347dea03649e6db1ed448ca1a4470a4cf9a26ee871"} Jul 24 15:54:39.688587 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:39.688560 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-96vtg" event={"ID":"6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a","Type":"ContainerStarted","Data":"1f4f5848b1b72716a69f0e8adca5b467c75ecb532fdab92b6bb73750d3304641"} Jul 24 15:54:39.690008 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:39.689987 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ggdsf" event={"ID":"ed6ff5c0-d89f-4154-8ff3-e365551e12da","Type":"ContainerStarted","Data":"ad85b6634b85293d5e299feff504879eca536f3ba78301a71bcdc4c23bcf1ecc"} Jul 24 15:54:39.690145 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:39.690126 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-ggdsf" Jul 24 15:54:39.704388 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:39.704338 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-4bw9l" podStartSLOduration=1.970463946 podStartE2EDuration="5.704321642s" totalImagesPullingTime="3.733857696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:54:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:54:39.7023833 +0000 UTC m=+554.620353599" watchObservedRunningTime="2026-07-24 15:54:39.704321642 +0000 UTC m=+554.622291942" Jul 24 15:54:39.718071 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:39.718014 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-ggdsf" podStartSLOduration=2.021361295 podStartE2EDuration="6.717993563s" totalImagesPullingTime="4.696632268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:54:39.71757774 +0000 UTC m=+554.635548042" watchObservedRunningTime="2026-07-24 15:54:39.717993563 +0000 UTC m=+554.635963864" Jul 24 15:54:39.737929 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:39.737890 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-96vtg"] Jul 24 15:54:39.746351 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:39.746297 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-96vtg" podStartSLOduration=1.7708664079999998 podStartE2EDuration="5.746277461s" totalImagesPullingTime="3.975411053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:54:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:54:39.744200326 +0000 UTC m=+554.662170660" watchObservedRunningTime="2026-07-24 15:54:39.746277461 +0000 UTC m=+554.664247760" Jul 24 15:54:41.701396 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:41.701361 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-96vtg" podUID="6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a" containerName="authorino" containerID="cri-o://1f4f5848b1b72716a69f0e8adca5b467c75ecb532fdab92b6bb73750d3304641" gracePeriod=30 Jul 24 15:54:41.958131 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:41.958067 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-96vtg" Jul 24 15:54:42.034576 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:42.034541 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4fwx\" (UniqueName: \"kubernetes.io/projected/6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a-kube-api-access-p4fwx\") pod \"6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a\" (UID: \"6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a\") " Jul 24 15:54:42.036584 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:42.036560 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a-kube-api-access-p4fwx" (OuterVolumeSpecName: "kube-api-access-p4fwx") pod "6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a" (UID: "6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a"). InnerVolumeSpecName "kube-api-access-p4fwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:54:42.135814 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:42.135779 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4fwx\" (UniqueName: \"kubernetes.io/projected/6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a-kube-api-access-p4fwx\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:54:42.708350 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:42.708319 2569 generic.go:356] "Generic (PLEG): container finished" podID="6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a" containerID="1f4f5848b1b72716a69f0e8adca5b467c75ecb532fdab92b6bb73750d3304641" exitCode=0 Jul 24 15:54:42.708824 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:42.708372 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-96vtg" Jul 24 15:54:42.708824 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:42.708406 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-96vtg" event={"ID":"6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a","Type":"ContainerDied","Data":"1f4f5848b1b72716a69f0e8adca5b467c75ecb532fdab92b6bb73750d3304641"} Jul 24 15:54:42.708824 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:42.708455 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-96vtg" event={"ID":"6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a","Type":"ContainerDied","Data":"b038f0010f402e86c865e64c92e07bfed2494ada285329e7a1a5908d9e922112"} Jul 24 15:54:42.708824 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:42.708479 2569 scope.go:117] "RemoveContainer" containerID="1f4f5848b1b72716a69f0e8adca5b467c75ecb532fdab92b6bb73750d3304641" Jul 24 15:54:42.717724 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:42.717704 2569 scope.go:117] "RemoveContainer" containerID="1f4f5848b1b72716a69f0e8adca5b467c75ecb532fdab92b6bb73750d3304641" Jul 24 15:54:42.717984 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:42.717965 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4f5848b1b72716a69f0e8adca5b467c75ecb532fdab92b6bb73750d3304641\": container with ID starting with 1f4f5848b1b72716a69f0e8adca5b467c75ecb532fdab92b6bb73750d3304641 not found: ID does not exist" containerID="1f4f5848b1b72716a69f0e8adca5b467c75ecb532fdab92b6bb73750d3304641" Jul 24 15:54:42.718025 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:42.717993 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4f5848b1b72716a69f0e8adca5b467c75ecb532fdab92b6bb73750d3304641"} err="failed to get container status \"1f4f5848b1b72716a69f0e8adca5b467c75ecb532fdab92b6bb73750d3304641\": rpc error: code = NotFound desc = could not find container \"1f4f5848b1b72716a69f0e8adca5b467c75ecb532fdab92b6bb73750d3304641\": container with ID starting with 1f4f5848b1b72716a69f0e8adca5b467c75ecb532fdab92b6bb73750d3304641 not found: ID does not exist" Jul 24 15:54:42.730800 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:42.730774 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-96vtg"] Jul 24 15:54:42.733227 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:42.733195 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-96vtg"] Jul 24 15:54:43.627682 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:43.627649 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a" path="/var/lib/kubelet/pods/6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a/volumes" Jul 24 15:54:48.565401 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:48.565358 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ggdsf"] Jul 24 15:54:48.565861 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:48.565698 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-ggdsf" podUID="ed6ff5c0-d89f-4154-8ff3-e365551e12da" containerName="limitador" containerID="cri-o://ad85b6634b85293d5e299feff504879eca536f3ba78301a71bcdc4c23bcf1ecc" gracePeriod=30 Jul 24 15:54:48.566432 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:48.566351 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-ggdsf" Jul 24 15:54:49.520983 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:49.520959 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ggdsf" Jul 24 15:54:49.613469 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:49.613442 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8ntd\" (UniqueName: \"kubernetes.io/projected/ed6ff5c0-d89f-4154-8ff3-e365551e12da-kube-api-access-c8ntd\") pod \"ed6ff5c0-d89f-4154-8ff3-e365551e12da\" (UID: \"ed6ff5c0-d89f-4154-8ff3-e365551e12da\") " Jul 24 15:54:49.613940 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:49.613476 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed6ff5c0-d89f-4154-8ff3-e365551e12da-config-file\") pod \"ed6ff5c0-d89f-4154-8ff3-e365551e12da\" (UID: \"ed6ff5c0-d89f-4154-8ff3-e365551e12da\") " Jul 24 15:54:49.613940 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:49.613923 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6ff5c0-d89f-4154-8ff3-e365551e12da-config-file" (OuterVolumeSpecName: "config-file") pod "ed6ff5c0-d89f-4154-8ff3-e365551e12da" (UID: "ed6ff5c0-d89f-4154-8ff3-e365551e12da"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:54:49.615654 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:49.615632 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6ff5c0-d89f-4154-8ff3-e365551e12da-kube-api-access-c8ntd" (OuterVolumeSpecName: "kube-api-access-c8ntd") pod "ed6ff5c0-d89f-4154-8ff3-e365551e12da" (UID: "ed6ff5c0-d89f-4154-8ff3-e365551e12da"). InnerVolumeSpecName "kube-api-access-c8ntd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:54:49.714458 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:49.714413 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8ntd\" (UniqueName: \"kubernetes.io/projected/ed6ff5c0-d89f-4154-8ff3-e365551e12da-kube-api-access-c8ntd\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:54:49.714458 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:49.714454 2569 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed6ff5c0-d89f-4154-8ff3-e365551e12da-config-file\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:54:49.753105 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:49.753068 2569 generic.go:356] "Generic (PLEG): container finished" podID="ed6ff5c0-d89f-4154-8ff3-e365551e12da" containerID="ad85b6634b85293d5e299feff504879eca536f3ba78301a71bcdc4c23bcf1ecc" exitCode=0 Jul 24 15:54:49.753326 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:49.753117 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ggdsf" Jul 24 15:54:49.753326 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:49.753163 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ggdsf" event={"ID":"ed6ff5c0-d89f-4154-8ff3-e365551e12da","Type":"ContainerDied","Data":"ad85b6634b85293d5e299feff504879eca536f3ba78301a71bcdc4c23bcf1ecc"} Jul 24 15:54:49.753326 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:49.753256 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ggdsf" event={"ID":"ed6ff5c0-d89f-4154-8ff3-e365551e12da","Type":"ContainerDied","Data":"71145b6c8773c2d18e60e2528e49c34ddf63d43fa000664a7290010160105689"} Jul 24 15:54:49.753326 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:49.753288 2569 scope.go:117] "RemoveContainer" containerID="ad85b6634b85293d5e299feff504879eca536f3ba78301a71bcdc4c23bcf1ecc" Jul 24 15:54:49.762689 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:49.762672 2569 scope.go:117] "RemoveContainer" containerID="ad85b6634b85293d5e299feff504879eca536f3ba78301a71bcdc4c23bcf1ecc" Jul 24 15:54:49.762963 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:54:49.762945 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad85b6634b85293d5e299feff504879eca536f3ba78301a71bcdc4c23bcf1ecc\": container with ID starting with ad85b6634b85293d5e299feff504879eca536f3ba78301a71bcdc4c23bcf1ecc not found: ID does not exist" containerID="ad85b6634b85293d5e299feff504879eca536f3ba78301a71bcdc4c23bcf1ecc" Jul 24 15:54:49.763031 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:49.762971 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad85b6634b85293d5e299feff504879eca536f3ba78301a71bcdc4c23bcf1ecc"} err="failed to get container status \"ad85b6634b85293d5e299feff504879eca536f3ba78301a71bcdc4c23bcf1ecc\": rpc error: code = NotFound desc = could not find container \"ad85b6634b85293d5e299feff504879eca536f3ba78301a71bcdc4c23bcf1ecc\": container with ID starting with ad85b6634b85293d5e299feff504879eca536f3ba78301a71bcdc4c23bcf1ecc not found: ID does not exist" Jul 24 15:54:49.772790 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:49.772760 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ggdsf"] Jul 24 15:54:49.775898 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:49.775875 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ggdsf"] Jul 24 15:54:50.132378 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.132344 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wwgfx"] Jul 24 15:54:50.132857 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.132840 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed6ff5c0-d89f-4154-8ff3-e365551e12da" containerName="limitador" Jul 24 15:54:50.132955 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.132861 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed6ff5c0-d89f-4154-8ff3-e365551e12da" containerName="limitador" Jul 24 15:54:50.132955 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.132888 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a" containerName="authorino" Jul 24 15:54:50.132955 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.132896 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a" containerName="authorino" Jul 24 15:54:50.133107 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.133027 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a8bf3fc-fb2d-4606-9ee4-3598ee63d45a" containerName="authorino" Jul 24 15:54:50.133107 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.133039 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed6ff5c0-d89f-4154-8ff3-e365551e12da" containerName="limitador" Jul 24 15:54:50.137009 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.136989 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwgfx" Jul 24 15:54:50.139953 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.139931 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 15:54:50.140070 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.139966 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 24 15:54:50.140070 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.139965 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 24 15:54:50.140952 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.140937 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jpdhq\"" Jul 24 15:54:50.145574 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.145555 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wwgfx"] Jul 24 15:54:50.218585 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.218552 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzfvb\" (UniqueName: \"kubernetes.io/projected/e71ea927-92d1-4cc4-9399-9c0a2148a8c6-kube-api-access-qzfvb\") pod \"postgres-69fb4594cf-wwgfx\" (UID: \"e71ea927-92d1-4cc4-9399-9c0a2148a8c6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwgfx" Jul 24 15:54:50.218765 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.218707 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e71ea927-92d1-4cc4-9399-9c0a2148a8c6-data\") pod \"postgres-69fb4594cf-wwgfx\" (UID: \"e71ea927-92d1-4cc4-9399-9c0a2148a8c6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwgfx" Jul 24 15:54:50.320142 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.320103 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e71ea927-92d1-4cc4-9399-9c0a2148a8c6-data\") pod \"postgres-69fb4594cf-wwgfx\" (UID: \"e71ea927-92d1-4cc4-9399-9c0a2148a8c6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwgfx" Jul 24 15:54:50.320343 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.320159 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzfvb\" (UniqueName: \"kubernetes.io/projected/e71ea927-92d1-4cc4-9399-9c0a2148a8c6-kube-api-access-qzfvb\") pod \"postgres-69fb4594cf-wwgfx\" (UID: \"e71ea927-92d1-4cc4-9399-9c0a2148a8c6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwgfx" Jul 24 15:54:50.320597 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.320580 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e71ea927-92d1-4cc4-9399-9c0a2148a8c6-data\") pod \"postgres-69fb4594cf-wwgfx\" (UID: \"e71ea927-92d1-4cc4-9399-9c0a2148a8c6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwgfx" Jul 24 15:54:50.329222 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.329179 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzfvb\" (UniqueName: \"kubernetes.io/projected/e71ea927-92d1-4cc4-9399-9c0a2148a8c6-kube-api-access-qzfvb\") pod \"postgres-69fb4594cf-wwgfx\" (UID: \"e71ea927-92d1-4cc4-9399-9c0a2148a8c6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwgfx" Jul 24 15:54:50.448550 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.448520 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwgfx" Jul 24 15:54:50.583424 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.583395 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wwgfx"] Jul 24 15:54:50.584607 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:54:50.584582 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71ea927_92d1_4cc4_9399_9c0a2148a8c6.slice/crio-fb434d8b0dfcb714c896991a17baeab9db48bec4f97920544c09b56df3b4856f WatchSource:0}: Error finding container fb434d8b0dfcb714c896991a17baeab9db48bec4f97920544c09b56df3b4856f: Status 404 returned error can't find the container with id fb434d8b0dfcb714c896991a17baeab9db48bec4f97920544c09b56df3b4856f Jul 24 15:54:50.760427 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:50.760345 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwgfx" event={"ID":"e71ea927-92d1-4cc4-9399-9c0a2148a8c6","Type":"ContainerStarted","Data":"fb434d8b0dfcb714c896991a17baeab9db48bec4f97920544c09b56df3b4856f"} Jul 24 15:54:51.633548 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:51.633509 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed6ff5c0-d89f-4154-8ff3-e365551e12da" path="/var/lib/kubelet/pods/ed6ff5c0-d89f-4154-8ff3-e365551e12da/volumes" Jul 24 15:54:55.801159 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:55.801125 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwgfx" event={"ID":"e71ea927-92d1-4cc4-9399-9c0a2148a8c6","Type":"ContainerStarted","Data":"96a8f34f6bc870817721661ad5a4e498b71d851ecb7e687d7a2fdff532b61821"} Jul 24 15:54:55.801554 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:55.801244 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwgfx" Jul 24 15:54:55.821468 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:54:55.821424 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwgfx" podStartSLOduration=0.709237834 podStartE2EDuration="5.821406917s" totalImagesPullingTime="5.112169083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:54:55.819598842 +0000 UTC m=+570.737569143" watchObservedRunningTime="2026-07-24 15:54:55.821406917 +0000 UTC m=+570.739377217" Jul 24 15:55:01.835937 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:01.835901 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wwgfx" Jul 24 15:55:02.397578 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:02.397544 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mr792"] Jul 24 15:55:02.403932 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:02.403911 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mr792" Jul 24 15:55:02.412470 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:02.412445 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mr792"] Jul 24 15:55:02.436728 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:02.436700 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2sw2\" (UniqueName: \"kubernetes.io/projected/5d042225-4faa-447e-8c87-110129c0c8ed-kube-api-access-r2sw2\") pod \"authorino-5d4b67c446-mr792\" (UID: \"5d042225-4faa-447e-8c87-110129c0c8ed\") " pod="kuadrant-system/authorino-5d4b67c446-mr792" Jul 24 15:55:02.537584 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:02.537547 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2sw2\" (UniqueName: \"kubernetes.io/projected/5d042225-4faa-447e-8c87-110129c0c8ed-kube-api-access-r2sw2\") pod \"authorino-5d4b67c446-mr792\" (UID: \"5d042225-4faa-447e-8c87-110129c0c8ed\") " pod="kuadrant-system/authorino-5d4b67c446-mr792" Jul 24 15:55:02.547315 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:02.547285 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2sw2\" (UniqueName: \"kubernetes.io/projected/5d042225-4faa-447e-8c87-110129c0c8ed-kube-api-access-r2sw2\") pod \"authorino-5d4b67c446-mr792\" (UID: \"5d042225-4faa-447e-8c87-110129c0c8ed\") " pod="kuadrant-system/authorino-5d4b67c446-mr792" Jul 24 15:55:02.650401 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:02.650309 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mr792"] Jul 24 15:55:02.650572 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:02.650497 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mr792" Jul 24 15:55:02.789167 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:02.789143 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mr792"] Jul 24 15:55:02.791039 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:55:02.791007 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d042225_4faa_447e_8c87_110129c0c8ed.slice/crio-20f7b1a0417c2662257ea643e508e042f0fe4709010e9034e74aad18e456c8ad WatchSource:0}: Error finding container 20f7b1a0417c2662257ea643e508e042f0fe4709010e9034e74aad18e456c8ad: Status 404 returned error can't find the container with id 20f7b1a0417c2662257ea643e508e042f0fe4709010e9034e74aad18e456c8ad Jul 24 15:55:02.841927 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:02.841892 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mr792" event={"ID":"5d042225-4faa-447e-8c87-110129c0c8ed","Type":"ContainerStarted","Data":"20f7b1a0417c2662257ea643e508e042f0fe4709010e9034e74aad18e456c8ad"} Jul 24 15:55:03.849397 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:03.849353 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mr792" event={"ID":"5d042225-4faa-447e-8c87-110129c0c8ed","Type":"ContainerStarted","Data":"a125af92227cce90145b35d0c0514e64c8c71677eb85d8276c3e4840ca9be038"} Jul 24 15:55:03.849881 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:03.849419 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-mr792" podUID="5d042225-4faa-447e-8c87-110129c0c8ed" containerName="authorino" containerID="cri-o://a125af92227cce90145b35d0c0514e64c8c71677eb85d8276c3e4840ca9be038" gracePeriod=30 Jul 24 15:55:03.865752 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:03.865706 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-mr792" podStartSLOduration=1.499251051 podStartE2EDuration="1.865689724s" totalImagesPullingTime="366.438673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:55:03.865506625 +0000 UTC m=+578.783476925" watchObservedRunningTime="2026-07-24 15:55:03.865689724 +0000 UTC m=+578.783660025" Jul 24 15:55:04.109054 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:04.108986 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mr792" Jul 24 15:55:04.150538 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:04.150503 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2sw2\" (UniqueName: \"kubernetes.io/projected/5d042225-4faa-447e-8c87-110129c0c8ed-kube-api-access-r2sw2\") pod \"5d042225-4faa-447e-8c87-110129c0c8ed\" (UID: \"5d042225-4faa-447e-8c87-110129c0c8ed\") " Jul 24 15:55:04.152645 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:04.152612 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d042225-4faa-447e-8c87-110129c0c8ed-kube-api-access-r2sw2" (OuterVolumeSpecName: "kube-api-access-r2sw2") pod "5d042225-4faa-447e-8c87-110129c0c8ed" (UID: "5d042225-4faa-447e-8c87-110129c0c8ed"). InnerVolumeSpecName "kube-api-access-r2sw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:55:04.252120 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:04.252084 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2sw2\" (UniqueName: \"kubernetes.io/projected/5d042225-4faa-447e-8c87-110129c0c8ed-kube-api-access-r2sw2\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:55:04.856852 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:04.856817 2569 generic.go:356] "Generic (PLEG): container finished" podID="5d042225-4faa-447e-8c87-110129c0c8ed" containerID="a125af92227cce90145b35d0c0514e64c8c71677eb85d8276c3e4840ca9be038" exitCode=0 Jul 24 15:55:04.857318 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:04.856861 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mr792" Jul 24 15:55:04.857318 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:04.856906 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mr792" event={"ID":"5d042225-4faa-447e-8c87-110129c0c8ed","Type":"ContainerDied","Data":"a125af92227cce90145b35d0c0514e64c8c71677eb85d8276c3e4840ca9be038"} Jul 24 15:55:04.857318 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:04.856952 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mr792" event={"ID":"5d042225-4faa-447e-8c87-110129c0c8ed","Type":"ContainerDied","Data":"20f7b1a0417c2662257ea643e508e042f0fe4709010e9034e74aad18e456c8ad"} Jul 24 15:55:04.857318 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:04.856973 2569 scope.go:117] "RemoveContainer" containerID="a125af92227cce90145b35d0c0514e64c8c71677eb85d8276c3e4840ca9be038" Jul 24 15:55:04.872490 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:04.872465 2569 scope.go:117] "RemoveContainer" containerID="a125af92227cce90145b35d0c0514e64c8c71677eb85d8276c3e4840ca9be038" Jul 24 15:55:04.872800 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:55:04.872780 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a125af92227cce90145b35d0c0514e64c8c71677eb85d8276c3e4840ca9be038\": container with ID starting with a125af92227cce90145b35d0c0514e64c8c71677eb85d8276c3e4840ca9be038 not found: ID does not exist" containerID="a125af92227cce90145b35d0c0514e64c8c71677eb85d8276c3e4840ca9be038" Jul 24 15:55:04.872878 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:04.872811 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a125af92227cce90145b35d0c0514e64c8c71677eb85d8276c3e4840ca9be038"} err="failed to get container status \"a125af92227cce90145b35d0c0514e64c8c71677eb85d8276c3e4840ca9be038\": rpc error: code = NotFound desc = could not find container \"a125af92227cce90145b35d0c0514e64c8c71677eb85d8276c3e4840ca9be038\": container with ID starting with a125af92227cce90145b35d0c0514e64c8c71677eb85d8276c3e4840ca9be038 not found: ID does not exist" Jul 24 15:55:04.886045 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:04.886015 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mr792"] Jul 24 15:55:04.893116 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:04.893091 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mr792"] Jul 24 15:55:05.628185 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:05.628151 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d042225-4faa-447e-8c87-110129c0c8ed" path="/var/lib/kubelet/pods/5d042225-4faa-447e-8c87-110129c0c8ed/volumes" Jul 24 15:55:06.728667 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:06.728635 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4bw9l"] Jul 24 15:55:06.729113 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:06.728814 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-4bw9l" podUID="ab904abe-80d4-4e21-a944-383fb364f361" containerName="authorino" containerID="cri-o://3657db68e3a45e8b9c58f9347dea03649e6db1ed448ca1a4470a4cf9a26ee871" gracePeriod=30 Jul 24 15:55:06.875057 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:06.875022 2569 generic.go:356] "Generic (PLEG): container finished" podID="ab904abe-80d4-4e21-a944-383fb364f361" containerID="3657db68e3a45e8b9c58f9347dea03649e6db1ed448ca1a4470a4cf9a26ee871" exitCode=0 Jul 24 15:55:06.875252 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:06.875096 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4bw9l" event={"ID":"ab904abe-80d4-4e21-a944-383fb364f361","Type":"ContainerDied","Data":"3657db68e3a45e8b9c58f9347dea03649e6db1ed448ca1a4470a4cf9a26ee871"} Jul 24 15:55:07.538896 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:07.538863 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4bw9l" Jul 24 15:55:07.583845 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:07.583816 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4n7w\" (UniqueName: \"kubernetes.io/projected/ab904abe-80d4-4e21-a944-383fb364f361-kube-api-access-j4n7w\") pod \"ab904abe-80d4-4e21-a944-383fb364f361\" (UID: \"ab904abe-80d4-4e21-a944-383fb364f361\") " Jul 24 15:55:07.585958 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:07.585930 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab904abe-80d4-4e21-a944-383fb364f361-kube-api-access-j4n7w" (OuterVolumeSpecName: "kube-api-access-j4n7w") pod "ab904abe-80d4-4e21-a944-383fb364f361" (UID: "ab904abe-80d4-4e21-a944-383fb364f361"). InnerVolumeSpecName "kube-api-access-j4n7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:55:07.684767 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:07.684732 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4n7w\" (UniqueName: \"kubernetes.io/projected/ab904abe-80d4-4e21-a944-383fb364f361-kube-api-access-j4n7w\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:55:07.882645 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:07.882607 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4bw9l" event={"ID":"ab904abe-80d4-4e21-a944-383fb364f361","Type":"ContainerDied","Data":"049d2a91a9e96759b1bc6822c3871b82eeaf7c0b64452ba80e8e76d0a04751ac"} Jul 24 15:55:07.883077 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:07.882663 2569 scope.go:117] "RemoveContainer" containerID="3657db68e3a45e8b9c58f9347dea03649e6db1ed448ca1a4470a4cf9a26ee871" Jul 24 15:55:07.883077 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:07.882624 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4bw9l" Jul 24 15:55:07.907359 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:07.907324 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4bw9l"] Jul 24 15:55:07.912244 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:07.912200 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4bw9l"] Jul 24 15:55:09.633960 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:09.633924 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab904abe-80d4-4e21-a944-383fb364f361" path="/var/lib/kubelet/pods/ab904abe-80d4-4e21-a944-383fb364f361/volumes" Jul 24 15:55:11.629175 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.629145 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-568994ddcf-gjxvs"] Jul 24 15:55:11.629573 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.629560 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d042225-4faa-447e-8c87-110129c0c8ed" containerName="authorino" Jul 24 15:55:11.629573 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.629571 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d042225-4faa-447e-8c87-110129c0c8ed" containerName="authorino" Jul 24 15:55:11.629646 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.629584 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab904abe-80d4-4e21-a944-383fb364f361" containerName="authorino" Jul 24 15:55:11.629646 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.629590 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab904abe-80d4-4e21-a944-383fb364f361" containerName="authorino" Jul 24 15:55:11.629707 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.629656 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab904abe-80d4-4e21-a944-383fb364f361" containerName="authorino" Jul 24 15:55:11.629707 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.629665 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d042225-4faa-447e-8c87-110129c0c8ed" containerName="authorino" Jul 24 15:55:11.637259 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.637238 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-568994ddcf-gjxvs" Jul 24 15:55:11.640701 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.640680 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 15:55:11.640826 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.640706 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 15:55:11.641740 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.641716 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4hsv9\"" Jul 24 15:55:11.648408 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.648385 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-568994ddcf-gjxvs"] Jul 24 15:55:11.724619 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.724583 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc6ct\" (UniqueName: \"kubernetes.io/projected/69879cd9-7d00-4294-8cab-8dd1903f08da-kube-api-access-lc6ct\") pod \"maas-controller-568994ddcf-gjxvs\" (UID: \"69879cd9-7d00-4294-8cab-8dd1903f08da\") " pod="opendatahub/maas-controller-568994ddcf-gjxvs" Jul 24 15:55:11.724816 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.724632 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69879cd9-7d00-4294-8cab-8dd1903f08da-webhook-cert\") pod \"maas-controller-568994ddcf-gjxvs\" (UID: \"69879cd9-7d00-4294-8cab-8dd1903f08da\") " pod="opendatahub/maas-controller-568994ddcf-gjxvs" Jul 24 15:55:11.796581 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.796533 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-685b879f45-4fxhf"] Jul 24 15:55:11.799284 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.799263 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685b879f45-4fxhf" Jul 24 15:55:11.810818 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.810793 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-685b879f45-4fxhf"] Jul 24 15:55:11.825827 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.825800 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc6ct\" (UniqueName: \"kubernetes.io/projected/69879cd9-7d00-4294-8cab-8dd1903f08da-kube-api-access-lc6ct\") pod \"maas-controller-568994ddcf-gjxvs\" (UID: \"69879cd9-7d00-4294-8cab-8dd1903f08da\") " pod="opendatahub/maas-controller-568994ddcf-gjxvs" Jul 24 15:55:11.825951 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.825839 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69879cd9-7d00-4294-8cab-8dd1903f08da-webhook-cert\") pod \"maas-controller-568994ddcf-gjxvs\" (UID: \"69879cd9-7d00-4294-8cab-8dd1903f08da\") " pod="opendatahub/maas-controller-568994ddcf-gjxvs" Jul 24 15:55:11.828296 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.828271 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69879cd9-7d00-4294-8cab-8dd1903f08da-webhook-cert\") pod \"maas-controller-568994ddcf-gjxvs\" (UID: \"69879cd9-7d00-4294-8cab-8dd1903f08da\") " pod="opendatahub/maas-controller-568994ddcf-gjxvs" Jul 24 15:55:11.834123 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.834098 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc6ct\" (UniqueName: \"kubernetes.io/projected/69879cd9-7d00-4294-8cab-8dd1903f08da-kube-api-access-lc6ct\") pod \"maas-controller-568994ddcf-gjxvs\" (UID: \"69879cd9-7d00-4294-8cab-8dd1903f08da\") " pod="opendatahub/maas-controller-568994ddcf-gjxvs" Jul 24 15:55:11.926998 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.926959 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ddf1a53-e905-4dd3-9d22-2f174196e1eb-webhook-cert\") pod \"maas-controller-685b879f45-4fxhf\" (UID: \"8ddf1a53-e905-4dd3-9d22-2f174196e1eb\") " pod="opendatahub/maas-controller-685b879f45-4fxhf" Jul 24 15:55:11.927176 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.927028 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxv2c\" (UniqueName: \"kubernetes.io/projected/8ddf1a53-e905-4dd3-9d22-2f174196e1eb-kube-api-access-xxv2c\") pod \"maas-controller-685b879f45-4fxhf\" (UID: \"8ddf1a53-e905-4dd3-9d22-2f174196e1eb\") " pod="opendatahub/maas-controller-685b879f45-4fxhf" Jul 24 15:55:11.948838 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:11.948808 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-568994ddcf-gjxvs" Jul 24 15:55:12.027748 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:12.027680 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ddf1a53-e905-4dd3-9d22-2f174196e1eb-webhook-cert\") pod \"maas-controller-685b879f45-4fxhf\" (UID: \"8ddf1a53-e905-4dd3-9d22-2f174196e1eb\") " pod="opendatahub/maas-controller-685b879f45-4fxhf" Jul 24 15:55:12.027939 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:12.027807 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxv2c\" (UniqueName: \"kubernetes.io/projected/8ddf1a53-e905-4dd3-9d22-2f174196e1eb-kube-api-access-xxv2c\") pod \"maas-controller-685b879f45-4fxhf\" (UID: \"8ddf1a53-e905-4dd3-9d22-2f174196e1eb\") " pod="opendatahub/maas-controller-685b879f45-4fxhf" Jul 24 15:55:12.030888 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:12.030860 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ddf1a53-e905-4dd3-9d22-2f174196e1eb-webhook-cert\") pod \"maas-controller-685b879f45-4fxhf\" (UID: \"8ddf1a53-e905-4dd3-9d22-2f174196e1eb\") " pod="opendatahub/maas-controller-685b879f45-4fxhf" Jul 24 15:55:12.037134 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:12.037105 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxv2c\" (UniqueName: \"kubernetes.io/projected/8ddf1a53-e905-4dd3-9d22-2f174196e1eb-kube-api-access-xxv2c\") pod \"maas-controller-685b879f45-4fxhf\" (UID: \"8ddf1a53-e905-4dd3-9d22-2f174196e1eb\") " pod="opendatahub/maas-controller-685b879f45-4fxhf" Jul 24 15:55:12.110190 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:12.110165 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685b879f45-4fxhf" Jul 24 15:55:12.111138 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:12.111111 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-568994ddcf-gjxvs"] Jul 24 15:55:12.112548 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:55:12.112524 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69879cd9_7d00_4294_8cab_8dd1903f08da.slice/crio-992130aff934b37d37d8d76b27af6c5830883c4f97bd7a0c0135506f44166133 WatchSource:0}: Error finding container 992130aff934b37d37d8d76b27af6c5830883c4f97bd7a0c0135506f44166133: Status 404 returned error can't find the container with id 992130aff934b37d37d8d76b27af6c5830883c4f97bd7a0c0135506f44166133 Jul 24 15:55:12.245077 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:12.245043 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-685b879f45-4fxhf"] Jul 24 15:55:12.246820 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:55:12.246792 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ddf1a53_e905_4dd3_9d22_2f174196e1eb.slice/crio-fe3674968cb04ed722251dcf24eb70fac0f289c5a54a90b5981192c67e3817ff WatchSource:0}: Error finding container fe3674968cb04ed722251dcf24eb70fac0f289c5a54a90b5981192c67e3817ff: Status 404 returned error can't find the container with id fe3674968cb04ed722251dcf24eb70fac0f289c5a54a90b5981192c67e3817ff Jul 24 15:55:12.919504 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:12.919418 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685b879f45-4fxhf" event={"ID":"8ddf1a53-e905-4dd3-9d22-2f174196e1eb","Type":"ContainerStarted","Data":"fe3674968cb04ed722251dcf24eb70fac0f289c5a54a90b5981192c67e3817ff"} Jul 24 15:55:12.923587 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:12.923523 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-568994ddcf-gjxvs" event={"ID":"69879cd9-7d00-4294-8cab-8dd1903f08da","Type":"ContainerStarted","Data":"992130aff934b37d37d8d76b27af6c5830883c4f97bd7a0c0135506f44166133"} Jul 24 15:55:14.945013 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:14.944966 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-568994ddcf-gjxvs" event={"ID":"69879cd9-7d00-4294-8cab-8dd1903f08da","Type":"ContainerStarted","Data":"6644d1a464b2cc3af93a3b395d17e5da7c8d46b4464cd92653a1362c103dfa04"} Jul 24 15:55:14.945486 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:14.945232 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-568994ddcf-gjxvs" Jul 24 15:55:14.946788 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:14.946764 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685b879f45-4fxhf" event={"ID":"8ddf1a53-e905-4dd3-9d22-2f174196e1eb","Type":"ContainerStarted","Data":"94d9a62c5590d65ea0b737735b562ed91745e7b3421a387f1b77e43932b3872a"} Jul 24 15:55:14.946920 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:14.946814 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-685b879f45-4fxhf" Jul 24 15:55:15.007337 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:15.007234 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-568994ddcf-gjxvs" podStartSLOduration=1.379064341 podStartE2EDuration="4.007194465s" totalImagesPullingTime="2.628130124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:55:15.003807549 +0000 UTC m=+589.921777849" watchObservedRunningTime="2026-07-24 15:55:15.007194465 +0000 UTC m=+589.925164766" Jul 24 15:55:15.041270 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:15.041194 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-685b879f45-4fxhf" podStartSLOduration=1.552555918 podStartE2EDuration="4.041179974s" totalImagesPullingTime="2.488624056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:55:15.038724087 +0000 UTC m=+589.956694392" watchObservedRunningTime="2026-07-24 15:55:15.041179974 +0000 UTC m=+589.959150275" Jul 24 15:55:25.471004 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:25.470974 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 15:55:25.476958 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:25.476939 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 15:55:25.477618 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:25.477596 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 15:55:25.483390 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:25.483371 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 15:55:25.955022 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:25.954995 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-568994ddcf-gjxvs" Jul 24 15:55:25.955194 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:25.955154 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-685b879f45-4fxhf" Jul 24 15:55:26.023743 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:26.023707 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-568994ddcf-gjxvs"] Jul 24 15:55:26.024065 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:26.023907 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-568994ddcf-gjxvs" podUID="69879cd9-7d00-4294-8cab-8dd1903f08da" containerName="manager" containerID="cri-o://6644d1a464b2cc3af93a3b395d17e5da7c8d46b4464cd92653a1362c103dfa04" gracePeriod=30 Jul 24 15:55:26.274534 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:26.274509 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-568994ddcf-gjxvs" Jul 24 15:55:26.370271 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:26.370237 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69879cd9-7d00-4294-8cab-8dd1903f08da-webhook-cert\") pod \"69879cd9-7d00-4294-8cab-8dd1903f08da\" (UID: \"69879cd9-7d00-4294-8cab-8dd1903f08da\") " Jul 24 15:55:26.370477 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:26.370291 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc6ct\" (UniqueName: \"kubernetes.io/projected/69879cd9-7d00-4294-8cab-8dd1903f08da-kube-api-access-lc6ct\") pod \"69879cd9-7d00-4294-8cab-8dd1903f08da\" (UID: \"69879cd9-7d00-4294-8cab-8dd1903f08da\") " Jul 24 15:55:26.372423 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:26.372394 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69879cd9-7d00-4294-8cab-8dd1903f08da-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "69879cd9-7d00-4294-8cab-8dd1903f08da" (UID: "69879cd9-7d00-4294-8cab-8dd1903f08da"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:55:26.372514 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:26.372464 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69879cd9-7d00-4294-8cab-8dd1903f08da-kube-api-access-lc6ct" (OuterVolumeSpecName: "kube-api-access-lc6ct") pod "69879cd9-7d00-4294-8cab-8dd1903f08da" (UID: "69879cd9-7d00-4294-8cab-8dd1903f08da"). InnerVolumeSpecName "kube-api-access-lc6ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:55:26.471612 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:26.471580 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc6ct\" (UniqueName: \"kubernetes.io/projected/69879cd9-7d00-4294-8cab-8dd1903f08da-kube-api-access-lc6ct\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:55:26.471612 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:26.471605 2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/69879cd9-7d00-4294-8cab-8dd1903f08da-webhook-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:55:27.026649 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:27.026618 2569 generic.go:356] "Generic (PLEG): container finished" podID="69879cd9-7d00-4294-8cab-8dd1903f08da" containerID="6644d1a464b2cc3af93a3b395d17e5da7c8d46b4464cd92653a1362c103dfa04" exitCode=0 Jul 24 15:55:27.026844 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:27.026655 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-568994ddcf-gjxvs" event={"ID":"69879cd9-7d00-4294-8cab-8dd1903f08da","Type":"ContainerDied","Data":"6644d1a464b2cc3af93a3b395d17e5da7c8d46b4464cd92653a1362c103dfa04"} Jul 24 15:55:27.026844 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:27.026661 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-568994ddcf-gjxvs" Jul 24 15:55:27.026844 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:27.026696 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-568994ddcf-gjxvs" event={"ID":"69879cd9-7d00-4294-8cab-8dd1903f08da","Type":"ContainerDied","Data":"992130aff934b37d37d8d76b27af6c5830883c4f97bd7a0c0135506f44166133"} Jul 24 15:55:27.026844 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:27.026718 2569 scope.go:117] "RemoveContainer" containerID="6644d1a464b2cc3af93a3b395d17e5da7c8d46b4464cd92653a1362c103dfa04" Jul 24 15:55:27.036808 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:27.036791 2569 scope.go:117] "RemoveContainer" containerID="6644d1a464b2cc3af93a3b395d17e5da7c8d46b4464cd92653a1362c103dfa04" Jul 24 15:55:27.037099 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:55:27.037076 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6644d1a464b2cc3af93a3b395d17e5da7c8d46b4464cd92653a1362c103dfa04\": container with ID starting with 6644d1a464b2cc3af93a3b395d17e5da7c8d46b4464cd92653a1362c103dfa04 not found: ID does not exist" containerID="6644d1a464b2cc3af93a3b395d17e5da7c8d46b4464cd92653a1362c103dfa04" Jul 24 15:55:27.037192 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:27.037122 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6644d1a464b2cc3af93a3b395d17e5da7c8d46b4464cd92653a1362c103dfa04"} err="failed to get container status \"6644d1a464b2cc3af93a3b395d17e5da7c8d46b4464cd92653a1362c103dfa04\": rpc error: code = NotFound desc = could not find container \"6644d1a464b2cc3af93a3b395d17e5da7c8d46b4464cd92653a1362c103dfa04\": container with ID starting with 6644d1a464b2cc3af93a3b395d17e5da7c8d46b4464cd92653a1362c103dfa04 not found: ID does not exist" Jul 24 15:55:27.049445 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:27.049421 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-568994ddcf-gjxvs"] Jul 24 15:55:27.054507 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:27.054484 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-568994ddcf-gjxvs"] Jul 24 15:55:27.633017 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:27.632980 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69879cd9-7d00-4294-8cab-8dd1903f08da" path="/var/lib/kubelet/pods/69879cd9-7d00-4294-8cab-8dd1903f08da/volumes" Jul 24 15:55:47.146980 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.146900 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-864cf74d58-9nc7m"] Jul 24 15:55:47.154653 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.147354 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69879cd9-7d00-4294-8cab-8dd1903f08da" containerName="manager" Jul 24 15:55:47.154653 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.147366 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="69879cd9-7d00-4294-8cab-8dd1903f08da" containerName="manager" Jul 24 15:55:47.154653 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.147441 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="69879cd9-7d00-4294-8cab-8dd1903f08da" containerName="manager" Jul 24 15:55:47.155157 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.155140 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-864cf74d58-9nc7m" Jul 24 15:55:47.164007 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.163985 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w4m8q\"" Jul 24 15:55:47.164171 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.163986 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 24 15:55:47.183539 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.183507 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-864cf74d58-9nc7m"] Jul 24 15:55:47.280496 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.280466 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49pbg\" (UniqueName: \"kubernetes.io/projected/611d9d18-c8e6-4cde-bbeb-559a55bea8cb-kube-api-access-49pbg\") pod \"maas-api-864cf74d58-9nc7m\" (UID: \"611d9d18-c8e6-4cde-bbeb-559a55bea8cb\") " pod="odh-ai-gateway-infra/maas-api-864cf74d58-9nc7m" Jul 24 15:55:47.280688 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.280512 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/611d9d18-c8e6-4cde-bbeb-559a55bea8cb-maas-api-tls\") pod \"maas-api-864cf74d58-9nc7m\" (UID: \"611d9d18-c8e6-4cde-bbeb-559a55bea8cb\") " pod="odh-ai-gateway-infra/maas-api-864cf74d58-9nc7m" Jul 24 15:55:47.280688 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.280553 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/611d9d18-c8e6-4cde-bbeb-559a55bea8cb-openshift-service-ca\") pod \"maas-api-864cf74d58-9nc7m\" (UID: \"611d9d18-c8e6-4cde-bbeb-559a55bea8cb\") " pod="odh-ai-gateway-infra/maas-api-864cf74d58-9nc7m" Jul 24 15:55:47.381138 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.381106 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49pbg\" (UniqueName: \"kubernetes.io/projected/611d9d18-c8e6-4cde-bbeb-559a55bea8cb-kube-api-access-49pbg\") pod \"maas-api-864cf74d58-9nc7m\" (UID: \"611d9d18-c8e6-4cde-bbeb-559a55bea8cb\") " pod="odh-ai-gateway-infra/maas-api-864cf74d58-9nc7m" Jul 24 15:55:47.381138 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.381147 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/611d9d18-c8e6-4cde-bbeb-559a55bea8cb-maas-api-tls\") pod \"maas-api-864cf74d58-9nc7m\" (UID: \"611d9d18-c8e6-4cde-bbeb-559a55bea8cb\") " pod="odh-ai-gateway-infra/maas-api-864cf74d58-9nc7m" Jul 24 15:55:47.381386 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.381167 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/611d9d18-c8e6-4cde-bbeb-559a55bea8cb-openshift-service-ca\") pod \"maas-api-864cf74d58-9nc7m\" (UID: \"611d9d18-c8e6-4cde-bbeb-559a55bea8cb\") " pod="odh-ai-gateway-infra/maas-api-864cf74d58-9nc7m" Jul 24 15:55:47.381963 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.381946 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/611d9d18-c8e6-4cde-bbeb-559a55bea8cb-openshift-service-ca\") pod \"maas-api-864cf74d58-9nc7m\" (UID: \"611d9d18-c8e6-4cde-bbeb-559a55bea8cb\") " pod="odh-ai-gateway-infra/maas-api-864cf74d58-9nc7m" Jul 24 15:55:47.383873 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.383846 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/611d9d18-c8e6-4cde-bbeb-559a55bea8cb-maas-api-tls\") pod \"maas-api-864cf74d58-9nc7m\" (UID: \"611d9d18-c8e6-4cde-bbeb-559a55bea8cb\") " pod="odh-ai-gateway-infra/maas-api-864cf74d58-9nc7m" Jul 24 15:55:47.397573 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.397510 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49pbg\" (UniqueName: \"kubernetes.io/projected/611d9d18-c8e6-4cde-bbeb-559a55bea8cb-kube-api-access-49pbg\") pod \"maas-api-864cf74d58-9nc7m\" (UID: \"611d9d18-c8e6-4cde-bbeb-559a55bea8cb\") " pod="odh-ai-gateway-infra/maas-api-864cf74d58-9nc7m" Jul 24 15:55:47.465828 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.465790 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-864cf74d58-9nc7m" Jul 24 15:55:47.630774 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.630749 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-864cf74d58-9nc7m"] Jul 24 15:55:47.632927 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:55:47.632899 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611d9d18_c8e6_4cde_bbeb_559a55bea8cb.slice/crio-c93e7c7c66eb82a81032af9185a66d91acb1fa616b5f2aababc50a3cee73f1b0 WatchSource:0}: Error finding container c93e7c7c66eb82a81032af9185a66d91acb1fa616b5f2aababc50a3cee73f1b0: Status 404 returned error can't find the container with id c93e7c7c66eb82a81032af9185a66d91acb1fa616b5f2aababc50a3cee73f1b0 Jul 24 15:55:47.634446 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:47.634424 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:55:48.155511 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:48.155477 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-864cf74d58-9nc7m" event={"ID":"611d9d18-c8e6-4cde-bbeb-559a55bea8cb","Type":"ContainerStarted","Data":"c93e7c7c66eb82a81032af9185a66d91acb1fa616b5f2aababc50a3cee73f1b0"} Jul 24 15:55:48.223505 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:48.223473 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-kbzx2"] Jul 24 15:55:48.229016 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:48.228991 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-kbzx2" Jul 24 15:55:48.233450 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:48.233428 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 24 15:55:48.233587 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:48.233506 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-fmtws\"" Jul 24 15:55:48.248664 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:48.248629 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-kbzx2"] Jul 24 15:55:48.292177 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:48.292135 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4fxj\" (UniqueName: \"kubernetes.io/projected/ca5f962c-6f6a-4360-874c-cadf65694353-kube-api-access-x4fxj\") pod \"payload-processing-5cc6cd444b-kbzx2\" (UID: \"ca5f962c-6f6a-4360-874c-cadf65694353\") " pod="openshift-ingress/payload-processing-5cc6cd444b-kbzx2" Jul 24 15:55:48.292380 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:48.292286 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5f962c-6f6a-4360-874c-cadf65694353-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-kbzx2\" (UID: \"ca5f962c-6f6a-4360-874c-cadf65694353\") " pod="openshift-ingress/payload-processing-5cc6cd444b-kbzx2" Jul 24 15:55:48.392841 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:48.392803 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4fxj\" (UniqueName: \"kubernetes.io/projected/ca5f962c-6f6a-4360-874c-cadf65694353-kube-api-access-x4fxj\") pod \"payload-processing-5cc6cd444b-kbzx2\" (UID: \"ca5f962c-6f6a-4360-874c-cadf65694353\") " pod="openshift-ingress/payload-processing-5cc6cd444b-kbzx2" Jul 24 15:55:48.393019 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:48.392868 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5f962c-6f6a-4360-874c-cadf65694353-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-kbzx2\" (UID: \"ca5f962c-6f6a-4360-874c-cadf65694353\") " pod="openshift-ingress/payload-processing-5cc6cd444b-kbzx2" Jul 24 15:55:48.393578 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:48.393557 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5f962c-6f6a-4360-874c-cadf65694353-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-kbzx2\" (UID: \"ca5f962c-6f6a-4360-874c-cadf65694353\") " pod="openshift-ingress/payload-processing-5cc6cd444b-kbzx2" Jul 24 15:55:48.409995 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:48.409921 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4fxj\" (UniqueName: \"kubernetes.io/projected/ca5f962c-6f6a-4360-874c-cadf65694353-kube-api-access-x4fxj\") pod \"payload-processing-5cc6cd444b-kbzx2\" (UID: \"ca5f962c-6f6a-4360-874c-cadf65694353\") " pod="openshift-ingress/payload-processing-5cc6cd444b-kbzx2" Jul 24 15:55:48.542021 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:48.541987 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-kbzx2" Jul 24 15:55:48.705944 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:48.705904 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-kbzx2"] Jul 24 15:55:48.708620 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:55:48.708579 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5f962c_6f6a_4360_874c_cadf65694353.slice/crio-07cca9796295dab8b4d4bfd7ad4abd76e69ea115972723f5758d5a565d8fe978 WatchSource:0}: Error finding container 07cca9796295dab8b4d4bfd7ad4abd76e69ea115972723f5758d5a565d8fe978: Status 404 returned error can't find the container with id 07cca9796295dab8b4d4bfd7ad4abd76e69ea115972723f5758d5a565d8fe978 Jul 24 15:55:49.164678 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:49.164640 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-kbzx2" event={"ID":"ca5f962c-6f6a-4360-874c-cadf65694353","Type":"ContainerStarted","Data":"07cca9796295dab8b4d4bfd7ad4abd76e69ea115972723f5758d5a565d8fe978"} Jul 24 15:55:50.172869 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:50.172835 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-864cf74d58-9nc7m" event={"ID":"611d9d18-c8e6-4cde-bbeb-559a55bea8cb","Type":"ContainerStarted","Data":"26f81373e1927234cbdb169ae1fc11de79c60fa84e83a62059d58584355728db"} Jul 24 15:55:50.173340 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:50.172927 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-864cf74d58-9nc7m" Jul 24 15:55:50.195368 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:50.195314 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-864cf74d58-9nc7m" podStartSLOduration=1.311739385 podStartE2EDuration="3.195289944s" totalImagesPullingTime="1.883550559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:55:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:55:50.193322719 +0000 UTC m=+625.111293025" watchObservedRunningTime="2026-07-24 15:55:50.195289944 +0000 UTC m=+625.113260245" Jul 24 15:55:51.180886 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:51.180844 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-kbzx2" event={"ID":"ca5f962c-6f6a-4360-874c-cadf65694353","Type":"ContainerStarted","Data":"52fa44f88ea52011eef88ec33132ce55f2cc0b3570815696a770045652fa8df5"} Jul 24 15:55:51.201937 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:51.201885 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-kbzx2" podStartSLOduration=1.317788366 podStartE2EDuration="3.201869633s" totalImagesPullingTime="1.884081267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:55:51.200887132 +0000 UTC m=+626.118857432" watchObservedRunningTime="2026-07-24 15:55:51.201869633 +0000 UTC m=+626.119839932" Jul 24 15:55:52.186540 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:52.186513 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-kbzx2" Jul 24 15:55:56.186292 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:55:56.186264 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-864cf74d58-9nc7m" Jul 24 15:56:03.195355 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:03.195324 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-kbzx2" Jul 24 15:56:15.382943 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.382911 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2"] Jul 24 15:56:15.388302 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.388280 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.391151 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.391126 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 15:56:15.391307 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.391156 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ngjlc\"" Jul 24 15:56:15.392750 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.392727 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 15:56:15.392878 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.392860 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 24 15:56:15.397723 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.397703 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2"] Jul 24 15:56:15.455379 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.455340 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13521c7f-020f-489d-b1dc-7996d725c55b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plv2\" (UID: \"13521c7f-020f-489d-b1dc-7996d725c55b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.455566 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.455387 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flrzq\" (UniqueName: \"kubernetes.io/projected/13521c7f-020f-489d-b1dc-7996d725c55b-kube-api-access-flrzq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plv2\" (UID: \"13521c7f-020f-489d-b1dc-7996d725c55b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.455566 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.455418 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13521c7f-020f-489d-b1dc-7996d725c55b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plv2\" (UID: \"13521c7f-020f-489d-b1dc-7996d725c55b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.455566 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.455439 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13521c7f-020f-489d-b1dc-7996d725c55b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plv2\" (UID: \"13521c7f-020f-489d-b1dc-7996d725c55b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.455734 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.455557 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13521c7f-020f-489d-b1dc-7996d725c55b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plv2\" (UID: \"13521c7f-020f-489d-b1dc-7996d725c55b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.455734 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.455697 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13521c7f-020f-489d-b1dc-7996d725c55b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plv2\" (UID: \"13521c7f-020f-489d-b1dc-7996d725c55b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.556867 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.556819 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13521c7f-020f-489d-b1dc-7996d725c55b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plv2\" (UID: \"13521c7f-020f-489d-b1dc-7996d725c55b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.556867 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.556870 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13521c7f-020f-489d-b1dc-7996d725c55b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plv2\" (UID: \"13521c7f-020f-489d-b1dc-7996d725c55b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.557103 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.556924 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13521c7f-020f-489d-b1dc-7996d725c55b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plv2\" (UID: \"13521c7f-020f-489d-b1dc-7996d725c55b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.557103 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.557001 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13521c7f-020f-489d-b1dc-7996d725c55b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plv2\" (UID: \"13521c7f-020f-489d-b1dc-7996d725c55b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.557103 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.557024 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13521c7f-020f-489d-b1dc-7996d725c55b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plv2\" (UID: \"13521c7f-020f-489d-b1dc-7996d725c55b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.557103 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.557044 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flrzq\" (UniqueName: \"kubernetes.io/projected/13521c7f-020f-489d-b1dc-7996d725c55b-kube-api-access-flrzq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plv2\" (UID: \"13521c7f-020f-489d-b1dc-7996d725c55b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.557412 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.557382 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13521c7f-020f-489d-b1dc-7996d725c55b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plv2\" (UID: \"13521c7f-020f-489d-b1dc-7996d725c55b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.557486 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.557418 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13521c7f-020f-489d-b1dc-7996d725c55b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plv2\" (UID: \"13521c7f-020f-489d-b1dc-7996d725c55b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.557554 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.557531 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13521c7f-020f-489d-b1dc-7996d725c55b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plv2\" (UID: \"13521c7f-020f-489d-b1dc-7996d725c55b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.559463 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.559444 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13521c7f-020f-489d-b1dc-7996d725c55b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plv2\" (UID: \"13521c7f-020f-489d-b1dc-7996d725c55b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.559612 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.559594 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13521c7f-020f-489d-b1dc-7996d725c55b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plv2\" (UID: \"13521c7f-020f-489d-b1dc-7996d725c55b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.565760 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.565737 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flrzq\" (UniqueName: \"kubernetes.io/projected/13521c7f-020f-489d-b1dc-7996d725c55b-kube-api-access-flrzq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4plv2\" (UID: \"13521c7f-020f-489d-b1dc-7996d725c55b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.701062 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.701020 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:15.842335 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:15.842308 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2"] Jul 24 15:56:15.843862 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:56:15.843824 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13521c7f_020f_489d_b1dc_7996d725c55b.slice/crio-7bf37c83dc61510d55d9cb6f399cb67e98d70a194853e2aadc668fcc71f5c301 WatchSource:0}: Error finding container 7bf37c83dc61510d55d9cb6f399cb67e98d70a194853e2aadc668fcc71f5c301: Status 404 returned error can't find the container with id 7bf37c83dc61510d55d9cb6f399cb67e98d70a194853e2aadc668fcc71f5c301 Jul 24 15:56:16.343858 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:16.343814 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" event={"ID":"13521c7f-020f-489d-b1dc-7996d725c55b","Type":"ContainerStarted","Data":"7bf37c83dc61510d55d9cb6f399cb67e98d70a194853e2aadc668fcc71f5c301"} Jul 24 15:56:17.351906 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:17.351809 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" event={"ID":"13521c7f-020f-489d-b1dc-7996d725c55b","Type":"ContainerStarted","Data":"4d9cb50a844dca050671b0533a359c0e4511f4435f0ec5456666d2185f7d3c21"} Jul 24 15:56:17.373741 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:17.373693 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" podStartSLOduration=1.212928077 podStartE2EDuration="2.37367723s" totalImagesPullingTime="1.160749153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:56:17.372455762 +0000 UTC m=+652.290426063" watchObservedRunningTime="2026-07-24 15:56:17.37367723 +0000 UTC m=+652.291647529" Jul 24 15:56:24.081595 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.081561 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk"] Jul 24 15:56:24.088602 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.088568 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.092349 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.092322 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 24 15:56:24.095865 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.095831 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk"] Jul 24 15:56:24.140361 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.140320 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0917d03-ecd9-4ca9-ae85-edf96a32b40b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jqdrk\" (UID: \"d0917d03-ecd9-4ca9-ae85-edf96a32b40b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.140525 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.140418 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sb9q\" (UniqueName: \"kubernetes.io/projected/d0917d03-ecd9-4ca9-ae85-edf96a32b40b-kube-api-access-5sb9q\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jqdrk\" (UID: \"d0917d03-ecd9-4ca9-ae85-edf96a32b40b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.140525 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.140458 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0917d03-ecd9-4ca9-ae85-edf96a32b40b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jqdrk\" (UID: \"d0917d03-ecd9-4ca9-ae85-edf96a32b40b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.140525 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.140483 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0917d03-ecd9-4ca9-ae85-edf96a32b40b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jqdrk\" (UID: \"d0917d03-ecd9-4ca9-ae85-edf96a32b40b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.140525 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.140512 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0917d03-ecd9-4ca9-ae85-edf96a32b40b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jqdrk\" (UID: \"d0917d03-ecd9-4ca9-ae85-edf96a32b40b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.140683 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.140600 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0917d03-ecd9-4ca9-ae85-edf96a32b40b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jqdrk\" (UID: \"d0917d03-ecd9-4ca9-ae85-edf96a32b40b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.241410 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.241364 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0917d03-ecd9-4ca9-ae85-edf96a32b40b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jqdrk\" (UID: \"d0917d03-ecd9-4ca9-ae85-edf96a32b40b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.241604 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.241489 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sb9q\" (UniqueName: \"kubernetes.io/projected/d0917d03-ecd9-4ca9-ae85-edf96a32b40b-kube-api-access-5sb9q\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jqdrk\" (UID: \"d0917d03-ecd9-4ca9-ae85-edf96a32b40b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.241604 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.241531 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0917d03-ecd9-4ca9-ae85-edf96a32b40b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jqdrk\" (UID: \"d0917d03-ecd9-4ca9-ae85-edf96a32b40b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.241604 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.241571 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0917d03-ecd9-4ca9-ae85-edf96a32b40b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jqdrk\" (UID: \"d0917d03-ecd9-4ca9-ae85-edf96a32b40b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.241780 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.241619 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0917d03-ecd9-4ca9-ae85-edf96a32b40b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jqdrk\" (UID: \"d0917d03-ecd9-4ca9-ae85-edf96a32b40b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.241780 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.241665 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0917d03-ecd9-4ca9-ae85-edf96a32b40b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jqdrk\" (UID: \"d0917d03-ecd9-4ca9-ae85-edf96a32b40b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.241967 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.241932 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0917d03-ecd9-4ca9-ae85-edf96a32b40b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jqdrk\" (UID: \"d0917d03-ecd9-4ca9-ae85-edf96a32b40b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.242080 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.241988 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0917d03-ecd9-4ca9-ae85-edf96a32b40b-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jqdrk\" (UID: \"d0917d03-ecd9-4ca9-ae85-edf96a32b40b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.242080 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.242033 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0917d03-ecd9-4ca9-ae85-edf96a32b40b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jqdrk\" (UID: \"d0917d03-ecd9-4ca9-ae85-edf96a32b40b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.243986 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.243962 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0917d03-ecd9-4ca9-ae85-edf96a32b40b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jqdrk\" (UID: \"d0917d03-ecd9-4ca9-ae85-edf96a32b40b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.243986 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.243978 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0917d03-ecd9-4ca9-ae85-edf96a32b40b-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jqdrk\" (UID: \"d0917d03-ecd9-4ca9-ae85-edf96a32b40b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.250477 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.250456 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sb9q\" (UniqueName: \"kubernetes.io/projected/d0917d03-ecd9-4ca9-ae85-edf96a32b40b-kube-api-access-5sb9q\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jqdrk\" (UID: \"d0917d03-ecd9-4ca9-ae85-edf96a32b40b\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.403687 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.403654 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:24.554547 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:24.554522 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk"] Jul 24 15:56:24.555695 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:56:24.555662 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0917d03_ecd9_4ca9_ae85_edf96a32b40b.slice/crio-9d77af105c0536b19d8f4df4bca3692feb71588767781de848e9abad2c17ed75 WatchSource:0}: Error finding container 9d77af105c0536b19d8f4df4bca3692feb71588767781de848e9abad2c17ed75: Status 404 returned error can't find the container with id 9d77af105c0536b19d8f4df4bca3692feb71588767781de848e9abad2c17ed75 Jul 24 15:56:25.401271 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:25.401244 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" event={"ID":"d0917d03-ecd9-4ca9-ae85-edf96a32b40b","Type":"ContainerStarted","Data":"9d77af105c0536b19d8f4df4bca3692feb71588767781de848e9abad2c17ed75"} Jul 24 15:56:25.701364 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:25.701334 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:25.701542 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:25.701475 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:25.713941 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:25.713907 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:26.414705 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:26.414660 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" event={"ID":"d0917d03-ecd9-4ca9-ae85-edf96a32b40b","Type":"ContainerStarted","Data":"fceef34456d9776d620cad659a32c56aa46b7fece62fe004fb20c346eaebc40a"} Jul 24 15:56:26.427045 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:26.427011 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4plv2" Jul 24 15:56:26.435694 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:26.435636 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" podStartSLOduration=1.65388581 podStartE2EDuration="2.435620524s" totalImagesPullingTime="781.734714ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:56:26.433468508 +0000 UTC m=+661.351438810" watchObservedRunningTime="2026-07-24 15:56:26.435620524 +0000 UTC m=+661.353590824" Jul 24 15:56:34.403772 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:34.403731 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:34.404270 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:34.403822 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:34.416496 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:34.416470 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:56:34.481028 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:56:34.481002 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jqdrk" Jul 24 15:58:01.461301 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:01.461262 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78694cc8bf-m8l45"] Jul 24 15:58:01.465393 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:01.465368 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78694cc8bf-m8l45" Jul 24 15:58:01.467983 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:01.467959 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 24 15:58:01.469255 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:01.469232 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wqb4g\"" Jul 24 15:58:01.471418 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:01.471389 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78694cc8bf-m8l45"] Jul 24 15:58:01.574570 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:01.574536 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e5c23128-5563-4307-ae00-a054c6a39f1c-tls-cert\") pod \"authorino-78694cc8bf-m8l45\" (UID: \"e5c23128-5563-4307-ae00-a054c6a39f1c\") " pod="kuadrant-system/authorino-78694cc8bf-m8l45" Jul 24 15:58:01.574766 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:01.574634 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9m88\" (UniqueName: \"kubernetes.io/projected/e5c23128-5563-4307-ae00-a054c6a39f1c-kube-api-access-j9m88\") pod \"authorino-78694cc8bf-m8l45\" (UID: \"e5c23128-5563-4307-ae00-a054c6a39f1c\") " pod="kuadrant-system/authorino-78694cc8bf-m8l45" Jul 24 15:58:01.676117 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:01.676080 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e5c23128-5563-4307-ae00-a054c6a39f1c-tls-cert\") pod \"authorino-78694cc8bf-m8l45\" (UID: \"e5c23128-5563-4307-ae00-a054c6a39f1c\") " pod="kuadrant-system/authorino-78694cc8bf-m8l45" Jul 24 15:58:01.676117 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:01.676121 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9m88\" (UniqueName: \"kubernetes.io/projected/e5c23128-5563-4307-ae00-a054c6a39f1c-kube-api-access-j9m88\") pod \"authorino-78694cc8bf-m8l45\" (UID: \"e5c23128-5563-4307-ae00-a054c6a39f1c\") " pod="kuadrant-system/authorino-78694cc8bf-m8l45" Jul 24 15:58:01.678535 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:01.678506 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e5c23128-5563-4307-ae00-a054c6a39f1c-tls-cert\") pod \"authorino-78694cc8bf-m8l45\" (UID: \"e5c23128-5563-4307-ae00-a054c6a39f1c\") " pod="kuadrant-system/authorino-78694cc8bf-m8l45" Jul 24 15:58:01.685515 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:01.685496 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9m88\" (UniqueName: \"kubernetes.io/projected/e5c23128-5563-4307-ae00-a054c6a39f1c-kube-api-access-j9m88\") pod \"authorino-78694cc8bf-m8l45\" (UID: \"e5c23128-5563-4307-ae00-a054c6a39f1c\") " pod="kuadrant-system/authorino-78694cc8bf-m8l45" Jul 24 15:58:01.777298 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:01.777193 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78694cc8bf-m8l45" Jul 24 15:58:01.938943 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:01.938919 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78694cc8bf-m8l45"] Jul 24 15:58:01.941775 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:58:01.941747 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5c23128_5563_4307_ae00_a054c6a39f1c.slice/crio-e4221f8d362f5bb53ac6a9b7bb02248e6247d6e65e074337f07124bd952e2c75 WatchSource:0}: Error finding container e4221f8d362f5bb53ac6a9b7bb02248e6247d6e65e074337f07124bd952e2c75: Status 404 returned error can't find the container with id e4221f8d362f5bb53ac6a9b7bb02248e6247d6e65e074337f07124bd952e2c75 Jul 24 15:58:01.999459 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:01.999426 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78694cc8bf-m8l45" event={"ID":"e5c23128-5563-4307-ae00-a054c6a39f1c","Type":"ContainerStarted","Data":"e4221f8d362f5bb53ac6a9b7bb02248e6247d6e65e074337f07124bd952e2c75"} Jul 24 15:58:03.009855 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:03.009823 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78694cc8bf-m8l45" event={"ID":"e5c23128-5563-4307-ae00-a054c6a39f1c","Type":"ContainerStarted","Data":"9f57f3519bcf3a1fe4d12131deb0b54d306451a2ff604d94e4960d94eefd1949"} Jul 24 15:58:48.907201 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:48.907100 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-78694cc8bf-m8l45" podStartSLOduration=47.391476851 podStartE2EDuration="47.907082114s" totalImagesPullingTime="515.605263ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:58:03.050101854 +0000 UTC m=+757.968072155" watchObservedRunningTime="2026-07-24 15:58:48.907082114 +0000 UTC m=+803.825052466" Jul 24 15:58:48.908669 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:48.908631 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-685b879f45-4fxhf"] Jul 24 15:58:48.909103 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:48.909058 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-685b879f45-4fxhf" podUID="8ddf1a53-e905-4dd3-9d22-2f174196e1eb" containerName="manager" containerID="cri-o://94d9a62c5590d65ea0b737735b562ed91745e7b3421a387f1b77e43932b3872a" gracePeriod=30 Jul 24 15:58:49.183764 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:49.183736 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685b879f45-4fxhf" Jul 24 15:58:49.312298 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:49.312259 2569 generic.go:356] "Generic (PLEG): container finished" podID="8ddf1a53-e905-4dd3-9d22-2f174196e1eb" containerID="94d9a62c5590d65ea0b737735b562ed91745e7b3421a387f1b77e43932b3872a" exitCode=0 Jul 24 15:58:49.312298 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:49.312284 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685b879f45-4fxhf" Jul 24 15:58:49.312529 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:49.312319 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685b879f45-4fxhf" event={"ID":"8ddf1a53-e905-4dd3-9d22-2f174196e1eb","Type":"ContainerDied","Data":"94d9a62c5590d65ea0b737735b562ed91745e7b3421a387f1b77e43932b3872a"} Jul 24 15:58:49.312529 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:49.312386 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685b879f45-4fxhf" event={"ID":"8ddf1a53-e905-4dd3-9d22-2f174196e1eb","Type":"ContainerDied","Data":"fe3674968cb04ed722251dcf24eb70fac0f289c5a54a90b5981192c67e3817ff"} Jul 24 15:58:49.312529 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:49.312416 2569 scope.go:117] "RemoveContainer" containerID="94d9a62c5590d65ea0b737735b562ed91745e7b3421a387f1b77e43932b3872a" Jul 24 15:58:49.324205 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:49.324184 2569 scope.go:117] "RemoveContainer" containerID="94d9a62c5590d65ea0b737735b562ed91745e7b3421a387f1b77e43932b3872a" Jul 24 15:58:49.324591 ip-10-0-141-58 kubenswrapper[2569]: E0724 15:58:49.324562 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94d9a62c5590d65ea0b737735b562ed91745e7b3421a387f1b77e43932b3872a\": container with ID starting with 94d9a62c5590d65ea0b737735b562ed91745e7b3421a387f1b77e43932b3872a not found: ID does not exist" containerID="94d9a62c5590d65ea0b737735b562ed91745e7b3421a387f1b77e43932b3872a" Jul 24 15:58:49.324693 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:49.324603 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94d9a62c5590d65ea0b737735b562ed91745e7b3421a387f1b77e43932b3872a"} err="failed to get container status \"94d9a62c5590d65ea0b737735b562ed91745e7b3421a387f1b77e43932b3872a\": rpc error: code = NotFound desc = could not find container \"94d9a62c5590d65ea0b737735b562ed91745e7b3421a387f1b77e43932b3872a\": container with ID starting with 94d9a62c5590d65ea0b737735b562ed91745e7b3421a387f1b77e43932b3872a not found: ID does not exist" Jul 24 15:58:49.361171 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:49.361137 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxv2c\" (UniqueName: \"kubernetes.io/projected/8ddf1a53-e905-4dd3-9d22-2f174196e1eb-kube-api-access-xxv2c\") pod \"8ddf1a53-e905-4dd3-9d22-2f174196e1eb\" (UID: \"8ddf1a53-e905-4dd3-9d22-2f174196e1eb\") " Jul 24 15:58:49.361373 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:49.361262 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ddf1a53-e905-4dd3-9d22-2f174196e1eb-webhook-cert\") pod \"8ddf1a53-e905-4dd3-9d22-2f174196e1eb\" (UID: \"8ddf1a53-e905-4dd3-9d22-2f174196e1eb\") " Jul 24 15:58:49.363308 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:49.363274 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ddf1a53-e905-4dd3-9d22-2f174196e1eb-kube-api-access-xxv2c" (OuterVolumeSpecName: "kube-api-access-xxv2c") pod "8ddf1a53-e905-4dd3-9d22-2f174196e1eb" (UID: "8ddf1a53-e905-4dd3-9d22-2f174196e1eb"). InnerVolumeSpecName "kube-api-access-xxv2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:58:49.363456 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:49.363348 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ddf1a53-e905-4dd3-9d22-2f174196e1eb-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8ddf1a53-e905-4dd3-9d22-2f174196e1eb" (UID: "8ddf1a53-e905-4dd3-9d22-2f174196e1eb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:58:49.462749 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:49.462654 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxv2c\" (UniqueName: \"kubernetes.io/projected/8ddf1a53-e905-4dd3-9d22-2f174196e1eb-kube-api-access-xxv2c\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:58:49.462749 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:49.462696 2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ddf1a53-e905-4dd3-9d22-2f174196e1eb-webhook-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 15:58:49.637618 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:49.637591 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-685b879f45-4fxhf"] Jul 24 15:58:49.645035 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:49.645004 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-685b879f45-4fxhf"] Jul 24 15:58:51.628918 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:58:51.628882 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ddf1a53-e905-4dd3-9d22-2f174196e1eb" path="/var/lib/kubelet/pods/8ddf1a53-e905-4dd3-9d22-2f174196e1eb/volumes" Jul 24 15:59:50.450747 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:50.450712 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-685b879f45-flqfl"] Jul 24 15:59:50.451241 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:50.451225 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ddf1a53-e905-4dd3-9d22-2f174196e1eb" containerName="manager" Jul 24 15:59:50.451311 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:50.451242 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ddf1a53-e905-4dd3-9d22-2f174196e1eb" containerName="manager" Jul 24 15:59:50.451311 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:50.451309 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ddf1a53-e905-4dd3-9d22-2f174196e1eb" containerName="manager" Jul 24 15:59:50.454403 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:50.454386 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685b879f45-flqfl" Jul 24 15:59:50.457931 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:50.457904 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4hsv9\"" Jul 24 15:59:50.458063 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:50.457944 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 15:59:50.458063 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:50.457959 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 15:59:50.463247 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:50.463226 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-685b879f45-flqfl"] Jul 24 15:59:50.573658 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:50.573618 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/041b1d50-f676-4218-b859-952d5a9b78cc-webhook-cert\") pod \"maas-controller-685b879f45-flqfl\" (UID: \"041b1d50-f676-4218-b859-952d5a9b78cc\") " pod="opendatahub/maas-controller-685b879f45-flqfl" Jul 24 15:59:50.573658 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:50.573665 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxckk\" (UniqueName: \"kubernetes.io/projected/041b1d50-f676-4218-b859-952d5a9b78cc-kube-api-access-hxckk\") pod \"maas-controller-685b879f45-flqfl\" (UID: \"041b1d50-f676-4218-b859-952d5a9b78cc\") " pod="opendatahub/maas-controller-685b879f45-flqfl" Jul 24 15:59:50.674862 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:50.674817 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/041b1d50-f676-4218-b859-952d5a9b78cc-webhook-cert\") pod \"maas-controller-685b879f45-flqfl\" (UID: \"041b1d50-f676-4218-b859-952d5a9b78cc\") " pod="opendatahub/maas-controller-685b879f45-flqfl" Jul 24 15:59:50.674862 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:50.674873 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxckk\" (UniqueName: \"kubernetes.io/projected/041b1d50-f676-4218-b859-952d5a9b78cc-kube-api-access-hxckk\") pod \"maas-controller-685b879f45-flqfl\" (UID: \"041b1d50-f676-4218-b859-952d5a9b78cc\") " pod="opendatahub/maas-controller-685b879f45-flqfl" Jul 24 15:59:50.677348 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:50.677319 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/041b1d50-f676-4218-b859-952d5a9b78cc-webhook-cert\") pod \"maas-controller-685b879f45-flqfl\" (UID: \"041b1d50-f676-4218-b859-952d5a9b78cc\") " pod="opendatahub/maas-controller-685b879f45-flqfl" Jul 24 15:59:50.683449 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:50.683424 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxckk\" (UniqueName: \"kubernetes.io/projected/041b1d50-f676-4218-b859-952d5a9b78cc-kube-api-access-hxckk\") pod \"maas-controller-685b879f45-flqfl\" (UID: \"041b1d50-f676-4218-b859-952d5a9b78cc\") " pod="opendatahub/maas-controller-685b879f45-flqfl" Jul 24 15:59:50.765909 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:50.765815 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685b879f45-flqfl" Jul 24 15:59:50.909311 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:50.909283 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-685b879f45-flqfl"] Jul 24 15:59:50.910270 ip-10-0-141-58 kubenswrapper[2569]: W0724 15:59:50.910242 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041b1d50_f676_4218_b859_952d5a9b78cc.slice/crio-3151c3ff7924e98f160161fb9c0bc0473cf79ded566e07d6e96bcc91f485629b WatchSource:0}: Error finding container 3151c3ff7924e98f160161fb9c0bc0473cf79ded566e07d6e96bcc91f485629b: Status 404 returned error can't find the container with id 3151c3ff7924e98f160161fb9c0bc0473cf79ded566e07d6e96bcc91f485629b Jul 24 15:59:51.709416 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:51.709375 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685b879f45-flqfl" event={"ID":"041b1d50-f676-4218-b859-952d5a9b78cc","Type":"ContainerStarted","Data":"0e10ee3c3d7c52a15cfddfe2812ff0ee4d2eff0fde25a341ee748df2bca33435"} Jul 24 15:59:51.709866 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:51.709436 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685b879f45-flqfl" event={"ID":"041b1d50-f676-4218-b859-952d5a9b78cc","Type":"ContainerStarted","Data":"3151c3ff7924e98f160161fb9c0bc0473cf79ded566e07d6e96bcc91f485629b"} Jul 24 15:59:51.709866 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:51.709478 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-685b879f45-flqfl" Jul 24 15:59:51.728695 ip-10-0-141-58 kubenswrapper[2569]: I0724 15:59:51.728639 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-685b879f45-flqfl" podStartSLOduration=1.3732169490000001 podStartE2EDuration="1.728620186s" totalImagesPullingTime="355.403237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:59:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:59:51.727858424 +0000 UTC m=+866.645828724" watchObservedRunningTime="2026-07-24 15:59:51.728620186 +0000 UTC m=+866.646590487" Jul 24 16:00:02.718083 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:02.718054 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-685b879f45-flqfl" Jul 24 16:00:07.364905 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:07.364863 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-685b879f45-flqfl"] Jul 24 16:00:07.365383 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:07.365087 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-685b879f45-flqfl" podUID="041b1d50-f676-4218-b859-952d5a9b78cc" containerName="manager" containerID="cri-o://0e10ee3c3d7c52a15cfddfe2812ff0ee4d2eff0fde25a341ee748df2bca33435" gracePeriod=30 Jul 24 16:00:07.631501 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:07.631427 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685b879f45-flqfl" Jul 24 16:00:07.634564 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:07.634543 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxckk\" (UniqueName: \"kubernetes.io/projected/041b1d50-f676-4218-b859-952d5a9b78cc-kube-api-access-hxckk\") pod \"041b1d50-f676-4218-b859-952d5a9b78cc\" (UID: \"041b1d50-f676-4218-b859-952d5a9b78cc\") " Jul 24 16:00:07.634645 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:07.634600 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/041b1d50-f676-4218-b859-952d5a9b78cc-webhook-cert\") pod \"041b1d50-f676-4218-b859-952d5a9b78cc\" (UID: \"041b1d50-f676-4218-b859-952d5a9b78cc\") " Jul 24 16:00:07.636673 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:07.636638 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041b1d50-f676-4218-b859-952d5a9b78cc-kube-api-access-hxckk" (OuterVolumeSpecName: "kube-api-access-hxckk") pod "041b1d50-f676-4218-b859-952d5a9b78cc" (UID: "041b1d50-f676-4218-b859-952d5a9b78cc"). InnerVolumeSpecName "kube-api-access-hxckk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:00:07.636673 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:07.636661 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/041b1d50-f676-4218-b859-952d5a9b78cc-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "041b1d50-f676-4218-b859-952d5a9b78cc" (UID: "041b1d50-f676-4218-b859-952d5a9b78cc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:00:07.736121 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:07.736081 2569 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/041b1d50-f676-4218-b859-952d5a9b78cc-webhook-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:00:07.736121 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:07.736120 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxckk\" (UniqueName: \"kubernetes.io/projected/041b1d50-f676-4218-b859-952d5a9b78cc-kube-api-access-hxckk\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:00:07.813523 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:07.813488 2569 generic.go:356] "Generic (PLEG): container finished" podID="041b1d50-f676-4218-b859-952d5a9b78cc" containerID="0e10ee3c3d7c52a15cfddfe2812ff0ee4d2eff0fde25a341ee748df2bca33435" exitCode=0 Jul 24 16:00:07.813696 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:07.813530 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685b879f45-flqfl" Jul 24 16:00:07.813696 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:07.813574 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685b879f45-flqfl" event={"ID":"041b1d50-f676-4218-b859-952d5a9b78cc","Type":"ContainerDied","Data":"0e10ee3c3d7c52a15cfddfe2812ff0ee4d2eff0fde25a341ee748df2bca33435"} Jul 24 16:00:07.813696 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:07.813622 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685b879f45-flqfl" event={"ID":"041b1d50-f676-4218-b859-952d5a9b78cc","Type":"ContainerDied","Data":"3151c3ff7924e98f160161fb9c0bc0473cf79ded566e07d6e96bcc91f485629b"} Jul 24 16:00:07.813696 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:07.813644 2569 scope.go:117] "RemoveContainer" containerID="0e10ee3c3d7c52a15cfddfe2812ff0ee4d2eff0fde25a341ee748df2bca33435" Jul 24 16:00:07.823784 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:07.823767 2569 scope.go:117] "RemoveContainer" containerID="0e10ee3c3d7c52a15cfddfe2812ff0ee4d2eff0fde25a341ee748df2bca33435" Jul 24 16:00:07.824056 ip-10-0-141-58 kubenswrapper[2569]: E0724 16:00:07.824036 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e10ee3c3d7c52a15cfddfe2812ff0ee4d2eff0fde25a341ee748df2bca33435\": container with ID starting with 0e10ee3c3d7c52a15cfddfe2812ff0ee4d2eff0fde25a341ee748df2bca33435 not found: ID does not exist" containerID="0e10ee3c3d7c52a15cfddfe2812ff0ee4d2eff0fde25a341ee748df2bca33435" Jul 24 16:00:07.824125 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:07.824065 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e10ee3c3d7c52a15cfddfe2812ff0ee4d2eff0fde25a341ee748df2bca33435"} err="failed to get container status \"0e10ee3c3d7c52a15cfddfe2812ff0ee4d2eff0fde25a341ee748df2bca33435\": rpc error: code = NotFound desc = could not find container \"0e10ee3c3d7c52a15cfddfe2812ff0ee4d2eff0fde25a341ee748df2bca33435\": container with ID starting with 0e10ee3c3d7c52a15cfddfe2812ff0ee4d2eff0fde25a341ee748df2bca33435 not found: ID does not exist" Jul 24 16:00:07.836686 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:07.836657 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-685b879f45-flqfl"] Jul 24 16:00:07.839137 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:07.839116 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-685b879f45-flqfl"] Jul 24 16:00:08.606060 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:08.606027 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-685b879f45-qkclw"] Jul 24 16:00:08.606716 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:08.606695 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="041b1d50-f676-4218-b859-952d5a9b78cc" containerName="manager" Jul 24 16:00:08.606809 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:08.606718 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="041b1d50-f676-4218-b859-952d5a9b78cc" containerName="manager" Jul 24 16:00:08.606881 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:08.606868 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="041b1d50-f676-4218-b859-952d5a9b78cc" containerName="manager" Jul 24 16:00:08.611287 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:08.611264 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685b879f45-qkclw" Jul 24 16:00:08.614077 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:08.614058 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 16:00:08.615193 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:08.615174 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 16:00:08.615316 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:08.615173 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4hsv9\"" Jul 24 16:00:08.627500 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:08.626001 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-685b879f45-qkclw"] Jul 24 16:00:08.644493 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:08.644450 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e44bd61-85ea-4dc4-b1e1-b33e685d7b9a-webhook-cert\") pod \"maas-controller-685b879f45-qkclw\" (UID: \"2e44bd61-85ea-4dc4-b1e1-b33e685d7b9a\") " pod="opendatahub/maas-controller-685b879f45-qkclw" Jul 24 16:00:08.644697 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:08.644574 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qgg9\" (UniqueName: \"kubernetes.io/projected/2e44bd61-85ea-4dc4-b1e1-b33e685d7b9a-kube-api-access-6qgg9\") pod \"maas-controller-685b879f45-qkclw\" (UID: \"2e44bd61-85ea-4dc4-b1e1-b33e685d7b9a\") " pod="opendatahub/maas-controller-685b879f45-qkclw" Jul 24 16:00:08.745807 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:08.745763 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e44bd61-85ea-4dc4-b1e1-b33e685d7b9a-webhook-cert\") pod \"maas-controller-685b879f45-qkclw\" (UID: \"2e44bd61-85ea-4dc4-b1e1-b33e685d7b9a\") " pod="opendatahub/maas-controller-685b879f45-qkclw" Jul 24 16:00:08.746001 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:08.745900 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qgg9\" (UniqueName: \"kubernetes.io/projected/2e44bd61-85ea-4dc4-b1e1-b33e685d7b9a-kube-api-access-6qgg9\") pod \"maas-controller-685b879f45-qkclw\" (UID: \"2e44bd61-85ea-4dc4-b1e1-b33e685d7b9a\") " pod="opendatahub/maas-controller-685b879f45-qkclw" Jul 24 16:00:08.748229 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:08.748186 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2e44bd61-85ea-4dc4-b1e1-b33e685d7b9a-webhook-cert\") pod \"maas-controller-685b879f45-qkclw\" (UID: \"2e44bd61-85ea-4dc4-b1e1-b33e685d7b9a\") " pod="opendatahub/maas-controller-685b879f45-qkclw" Jul 24 16:00:08.757042 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:08.757017 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qgg9\" (UniqueName: \"kubernetes.io/projected/2e44bd61-85ea-4dc4-b1e1-b33e685d7b9a-kube-api-access-6qgg9\") pod \"maas-controller-685b879f45-qkclw\" (UID: \"2e44bd61-85ea-4dc4-b1e1-b33e685d7b9a\") " pod="opendatahub/maas-controller-685b879f45-qkclw" Jul 24 16:00:08.928431 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:08.928378 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-685b879f45-qkclw" Jul 24 16:00:09.071604 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:09.071572 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-685b879f45-qkclw"] Jul 24 16:00:09.072621 ip-10-0-141-58 kubenswrapper[2569]: W0724 16:00:09.072592 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e44bd61_85ea_4dc4_b1e1_b33e685d7b9a.slice/crio-fc3e0a5c28ab1bc056b96cd12371c70916f4b28aa8206ac80b2b1a7b7e138c5b WatchSource:0}: Error finding container fc3e0a5c28ab1bc056b96cd12371c70916f4b28aa8206ac80b2b1a7b7e138c5b: Status 404 returned error can't find the container with id fc3e0a5c28ab1bc056b96cd12371c70916f4b28aa8206ac80b2b1a7b7e138c5b Jul 24 16:00:09.630527 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:09.630494 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="041b1d50-f676-4218-b859-952d5a9b78cc" path="/var/lib/kubelet/pods/041b1d50-f676-4218-b859-952d5a9b78cc/volumes" Jul 24 16:00:09.829849 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:09.829753 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685b879f45-qkclw" event={"ID":"2e44bd61-85ea-4dc4-b1e1-b33e685d7b9a","Type":"ContainerStarted","Data":"714c398d0d1f6e443f7f57b6bd8270be325198f45f744ccfe399a9db19f700f7"} Jul 24 16:00:09.829849 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:09.829812 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-685b879f45-qkclw" event={"ID":"2e44bd61-85ea-4dc4-b1e1-b33e685d7b9a","Type":"ContainerStarted","Data":"fc3e0a5c28ab1bc056b96cd12371c70916f4b28aa8206ac80b2b1a7b7e138c5b"} Jul 24 16:00:09.830065 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:09.829952 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-685b879f45-qkclw" Jul 24 16:00:09.852620 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:09.852569 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-685b879f45-qkclw" podStartSLOduration=1.480749551 podStartE2EDuration="1.852554032s" totalImagesPullingTime="371.804481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:00:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:00:09.852272535 +0000 UTC m=+884.770242835" watchObservedRunningTime="2026-07-24 16:00:09.852554032 +0000 UTC m=+884.770524331" Jul 24 16:00:20.839268 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:20.839174 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-685b879f45-qkclw" Jul 24 16:00:25.516647 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:25.516616 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 16:00:25.520072 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:25.520050 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 16:00:25.522519 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:25.522501 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 16:00:25.526109 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:00:25.526091 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 16:04:50.068412 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:04:50.068329 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx"] Jul 24 16:04:50.068935 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:04:50.068560 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx" podUID="68b260d5-51be-44cb-a5ab-7c2e4e275e49" containerName="manager" containerID="cri-o://1d5b8100aaf1116cc5e799aa39dcd5c365d8b0d23385ebc9c2fca547d655de93" gracePeriod=10 Jul 24 16:04:52.937939 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:04:52.937916 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx" Jul 24 16:04:52.977521 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:04:52.977441 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68b260d5-51be-44cb-a5ab-7c2e4e275e49-extensions-socket-volume\") pod \"68b260d5-51be-44cb-a5ab-7c2e4e275e49\" (UID: \"68b260d5-51be-44cb-a5ab-7c2e4e275e49\") " Jul 24 16:04:52.977521 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:04:52.977484 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmhfh\" (UniqueName: \"kubernetes.io/projected/68b260d5-51be-44cb-a5ab-7c2e4e275e49-kube-api-access-jmhfh\") pod \"68b260d5-51be-44cb-a5ab-7c2e4e275e49\" (UID: \"68b260d5-51be-44cb-a5ab-7c2e4e275e49\") " Jul 24 16:04:52.977881 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:04:52.977856 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b260d5-51be-44cb-a5ab-7c2e4e275e49-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "68b260d5-51be-44cb-a5ab-7c2e4e275e49" (UID: "68b260d5-51be-44cb-a5ab-7c2e4e275e49"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:04:52.979788 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:04:52.979760 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b260d5-51be-44cb-a5ab-7c2e4e275e49-kube-api-access-jmhfh" (OuterVolumeSpecName: "kube-api-access-jmhfh") pod "68b260d5-51be-44cb-a5ab-7c2e4e275e49" (UID: "68b260d5-51be-44cb-a5ab-7c2e4e275e49"). InnerVolumeSpecName "kube-api-access-jmhfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:04:53.079008 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:04:53.078964 2569 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68b260d5-51be-44cb-a5ab-7c2e4e275e49-extensions-socket-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:04:53.079008 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:04:53.079005 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmhfh\" (UniqueName: \"kubernetes.io/projected/68b260d5-51be-44cb-a5ab-7c2e4e275e49-kube-api-access-jmhfh\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:04:53.613014 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:04:53.612981 2569 generic.go:356] "Generic (PLEG): container finished" podID="68b260d5-51be-44cb-a5ab-7c2e4e275e49" containerID="1d5b8100aaf1116cc5e799aa39dcd5c365d8b0d23385ebc9c2fca547d655de93" exitCode=0 Jul 24 16:04:53.613161 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:04:53.613023 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx" event={"ID":"68b260d5-51be-44cb-a5ab-7c2e4e275e49","Type":"ContainerDied","Data":"1d5b8100aaf1116cc5e799aa39dcd5c365d8b0d23385ebc9c2fca547d655de93"} Jul 24 16:04:53.613161 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:04:53.613033 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx" Jul 24 16:04:53.613161 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:04:53.613063 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx" event={"ID":"68b260d5-51be-44cb-a5ab-7c2e4e275e49","Type":"ContainerDied","Data":"e0e8ab3c716fe6d175a224eff1d8817c464d5bb1ccf1ac863145b78ad688e869"} Jul 24 16:04:53.613161 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:04:53.613084 2569 scope.go:117] "RemoveContainer" containerID="1d5b8100aaf1116cc5e799aa39dcd5c365d8b0d23385ebc9c2fca547d655de93" Jul 24 16:04:53.625016 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:04:53.624995 2569 scope.go:117] "RemoveContainer" containerID="1d5b8100aaf1116cc5e799aa39dcd5c365d8b0d23385ebc9c2fca547d655de93" Jul 24 16:04:53.625293 ip-10-0-141-58 kubenswrapper[2569]: E0724 16:04:53.625271 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d5b8100aaf1116cc5e799aa39dcd5c365d8b0d23385ebc9c2fca547d655de93\": container with ID starting with 1d5b8100aaf1116cc5e799aa39dcd5c365d8b0d23385ebc9c2fca547d655de93 not found: ID does not exist" containerID="1d5b8100aaf1116cc5e799aa39dcd5c365d8b0d23385ebc9c2fca547d655de93" Jul 24 16:04:53.625361 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:04:53.625302 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d5b8100aaf1116cc5e799aa39dcd5c365d8b0d23385ebc9c2fca547d655de93"} err="failed to get container status \"1d5b8100aaf1116cc5e799aa39dcd5c365d8b0d23385ebc9c2fca547d655de93\": rpc error: code = NotFound desc = could not find container \"1d5b8100aaf1116cc5e799aa39dcd5c365d8b0d23385ebc9c2fca547d655de93\": container with ID starting with 1d5b8100aaf1116cc5e799aa39dcd5c365d8b0d23385ebc9c2fca547d655de93 not found: ID does not exist" Jul 24 16:04:53.636654 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:04:53.636626 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx"] Jul 24 16:04:53.640319 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:04:53.640297 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7r4rx"] Jul 24 16:04:55.628726 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:04:55.628690 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68b260d5-51be-44cb-a5ab-7c2e4e275e49" path="/var/lib/kubelet/pods/68b260d5-51be-44cb-a5ab-7c2e4e275e49/volumes" Jul 24 16:05:25.560969 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:25.560933 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 16:05:25.565416 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:25.565393 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 16:05:25.566559 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:25.566541 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 16:05:25.580203 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:25.580172 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 16:05:51.254737 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:51.254694 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-v9kzl"] Jul 24 16:05:51.255421 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:51.255402 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68b260d5-51be-44cb-a5ab-7c2e4e275e49" containerName="manager" Jul 24 16:05:51.255537 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:51.255423 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="68b260d5-51be-44cb-a5ab-7c2e4e275e49" containerName="manager" Jul 24 16:05:51.255594 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:51.255568 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="68b260d5-51be-44cb-a5ab-7c2e4e275e49" containerName="manager" Jul 24 16:05:51.260048 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:51.260027 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-v9kzl" Jul 24 16:05:51.262563 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:51.262545 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6l4rd\"" Jul 24 16:05:51.268962 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:51.268935 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-v9kzl"] Jul 24 16:05:51.347396 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:51.347365 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzm2r\" (UniqueName: \"kubernetes.io/projected/b750b5de-3604-4e92-ac6f-12c31cf8d7fd-kube-api-access-vzm2r\") pod \"kuadrant-operator-controller-manager-86878c769f-v9kzl\" (UID: \"b750b5de-3604-4e92-ac6f-12c31cf8d7fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-v9kzl" Jul 24 16:05:51.347580 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:51.347467 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b750b5de-3604-4e92-ac6f-12c31cf8d7fd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-v9kzl\" (UID: \"b750b5de-3604-4e92-ac6f-12c31cf8d7fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-v9kzl" Jul 24 16:05:51.448919 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:51.448870 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzm2r\" (UniqueName: \"kubernetes.io/projected/b750b5de-3604-4e92-ac6f-12c31cf8d7fd-kube-api-access-vzm2r\") pod \"kuadrant-operator-controller-manager-86878c769f-v9kzl\" (UID: \"b750b5de-3604-4e92-ac6f-12c31cf8d7fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-v9kzl" Jul 24 16:05:51.449103 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:51.448958 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b750b5de-3604-4e92-ac6f-12c31cf8d7fd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-v9kzl\" (UID: \"b750b5de-3604-4e92-ac6f-12c31cf8d7fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-v9kzl" Jul 24 16:05:51.449331 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:51.449315 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b750b5de-3604-4e92-ac6f-12c31cf8d7fd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-v9kzl\" (UID: \"b750b5de-3604-4e92-ac6f-12c31cf8d7fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-v9kzl" Jul 24 16:05:51.462021 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:51.461993 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzm2r\" (UniqueName: \"kubernetes.io/projected/b750b5de-3604-4e92-ac6f-12c31cf8d7fd-kube-api-access-vzm2r\") pod \"kuadrant-operator-controller-manager-86878c769f-v9kzl\" (UID: \"b750b5de-3604-4e92-ac6f-12c31cf8d7fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-v9kzl" Jul 24 16:05:51.571126 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:51.571036 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-v9kzl" Jul 24 16:05:51.720624 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:51.720594 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-v9kzl"] Jul 24 16:05:51.721586 ip-10-0-141-58 kubenswrapper[2569]: W0724 16:05:51.721560 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb750b5de_3604_4e92_ac6f_12c31cf8d7fd.slice/crio-866970bb03258efaa60067b51d8cff0f5237026a0b3a198ef3a49559021aa06d WatchSource:0}: Error finding container 866970bb03258efaa60067b51d8cff0f5237026a0b3a198ef3a49559021aa06d: Status 404 returned error can't find the container with id 866970bb03258efaa60067b51d8cff0f5237026a0b3a198ef3a49559021aa06d Jul 24 16:05:51.976857 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:51.976820 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-v9kzl" event={"ID":"b750b5de-3604-4e92-ac6f-12c31cf8d7fd","Type":"ContainerStarted","Data":"9ba9312b58576cb1994144d2ca831663a249b1efc7d9d3b7a78f5d261de4e18c"} Jul 24 16:05:51.977042 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:51.976878 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-v9kzl" event={"ID":"b750b5de-3604-4e92-ac6f-12c31cf8d7fd","Type":"ContainerStarted","Data":"866970bb03258efaa60067b51d8cff0f5237026a0b3a198ef3a49559021aa06d"} Jul 24 16:05:51.977042 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:51.976997 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-v9kzl" Jul 24 16:05:51.995666 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:05:51.995616 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-v9kzl" podStartSLOduration=0.995600233 podStartE2EDuration="995.600233ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:05:51.994692785 +0000 UTC m=+1226.912663084" watchObservedRunningTime="2026-07-24 16:05:51.995600233 +0000 UTC m=+1226.913570600" Jul 24 16:06:02.985762 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:06:02.985730 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-v9kzl" Jul 24 16:10:25.609718 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:10:25.609693 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 16:10:25.615308 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:10:25.615289 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 16:10:25.617381 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:10:25.617361 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 16:10:25.623439 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:10:25.623419 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 16:13:15.844016 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:15.843983 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t"] Jul 24 16:13:15.848037 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:15.848016 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" Jul 24 16:13:15.850524 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:15.850501 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c3420287-serving-cert\"" Jul 24 16:13:15.856514 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:15.856486 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5931b2af-399f-4b0d-ac57-3ed52182ed04-openshift-service-ca\") pod \"maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t\" (UID: \"5931b2af-399f-4b0d-ac57-3ed52182ed04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" Jul 24 16:13:15.856775 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:15.856752 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptjz7\" (UniqueName: \"kubernetes.io/projected/5931b2af-399f-4b0d-ac57-3ed52182ed04-kube-api-access-ptjz7\") pod \"maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t\" (UID: \"5931b2af-399f-4b0d-ac57-3ed52182ed04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" Jul 24 16:13:15.856960 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:15.856791 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5931b2af-399f-4b0d-ac57-3ed52182ed04-maas-api-tls\") pod \"maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t\" (UID: \"5931b2af-399f-4b0d-ac57-3ed52182ed04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" Jul 24 16:13:15.858184 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:15.858163 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t"] Jul 24 16:13:15.957644 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:15.957614 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5931b2af-399f-4b0d-ac57-3ed52182ed04-openshift-service-ca\") pod \"maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t\" (UID: \"5931b2af-399f-4b0d-ac57-3ed52182ed04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" Jul 24 16:13:15.957844 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:15.957673 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptjz7\" (UniqueName: \"kubernetes.io/projected/5931b2af-399f-4b0d-ac57-3ed52182ed04-kube-api-access-ptjz7\") pod \"maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t\" (UID: \"5931b2af-399f-4b0d-ac57-3ed52182ed04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" Jul 24 16:13:15.957844 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:15.957694 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5931b2af-399f-4b0d-ac57-3ed52182ed04-maas-api-tls\") pod \"maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t\" (UID: \"5931b2af-399f-4b0d-ac57-3ed52182ed04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" Jul 24 16:13:15.957844 ip-10-0-141-58 kubenswrapper[2569]: E0724 16:13:15.957809 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-serving-cert: secret "maas-api-e2e-ait-c3420287-serving-cert" not found Jul 24 16:13:15.958031 ip-10-0-141-58 kubenswrapper[2569]: E0724 16:13:15.957866 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5931b2af-399f-4b0d-ac57-3ed52182ed04-maas-api-tls podName:5931b2af-399f-4b0d-ac57-3ed52182ed04 nodeName:}" failed. No retries permitted until 2026-07-24 16:13:16.457848746 +0000 UTC m=+1671.375819024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5931b2af-399f-4b0d-ac57-3ed52182ed04-maas-api-tls") pod "maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" (UID: "5931b2af-399f-4b0d-ac57-3ed52182ed04") : secret "maas-api-e2e-ait-c3420287-serving-cert" not found Jul 24 16:13:15.958404 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:15.958379 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5931b2af-399f-4b0d-ac57-3ed52182ed04-openshift-service-ca\") pod \"maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t\" (UID: \"5931b2af-399f-4b0d-ac57-3ed52182ed04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" Jul 24 16:13:15.983193 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:15.983164 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptjz7\" (UniqueName: \"kubernetes.io/projected/5931b2af-399f-4b0d-ac57-3ed52182ed04-kube-api-access-ptjz7\") pod \"maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t\" (UID: \"5931b2af-399f-4b0d-ac57-3ed52182ed04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" Jul 24 16:13:16.463258 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:16.463178 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5931b2af-399f-4b0d-ac57-3ed52182ed04-maas-api-tls\") pod \"maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t\" (UID: \"5931b2af-399f-4b0d-ac57-3ed52182ed04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" Jul 24 16:13:16.465961 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:16.465932 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5931b2af-399f-4b0d-ac57-3ed52182ed04-maas-api-tls\") pod \"maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t\" (UID: \"5931b2af-399f-4b0d-ac57-3ed52182ed04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" Jul 24 16:13:16.759722 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:16.759629 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" Jul 24 16:13:16.955590 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:16.955551 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t"] Jul 24 16:13:16.957152 ip-10-0-141-58 kubenswrapper[2569]: W0724 16:13:16.957125 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5931b2af_399f_4b0d_ac57_3ed52182ed04.slice/crio-61cba0771de21eabfb7981a1fe245a0d6b601cdfdc7e66988d5bce5bcf8c59e4 WatchSource:0}: Error finding container 61cba0771de21eabfb7981a1fe245a0d6b601cdfdc7e66988d5bce5bcf8c59e4: Status 404 returned error can't find the container with id 61cba0771de21eabfb7981a1fe245a0d6b601cdfdc7e66988d5bce5bcf8c59e4 Jul 24 16:13:16.958548 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:16.958529 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 16:13:17.813055 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:17.813018 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" event={"ID":"5931b2af-399f-4b0d-ac57-3ed52182ed04","Type":"ContainerStarted","Data":"c681fc6902713f034c8542802660df0893e98a42fbf6333b5cd972e1d4d95877"} Jul 24 16:13:17.813250 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:17.813078 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" Jul 24 16:13:17.813250 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:17.813093 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" event={"ID":"5931b2af-399f-4b0d-ac57-3ed52182ed04","Type":"ContainerStarted","Data":"61cba0771de21eabfb7981a1fe245a0d6b601cdfdc7e66988d5bce5bcf8c59e4"} Jul 24 16:13:17.853064 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:17.852951 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" podStartSLOduration=2.243670904 podStartE2EDuration="2.852936484s" totalImagesPullingTime="609.26558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:13:17.851111637 +0000 UTC m=+1672.769081939" watchObservedRunningTime="2026-07-24 16:13:17.852936484 +0000 UTC m=+1672.770906784" Jul 24 16:13:23.825713 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:23.825637 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" Jul 24 16:13:51.935242 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:51.935134 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t"] Jul 24 16:13:51.935806 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:51.935413 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" podUID="5931b2af-399f-4b0d-ac57-3ed52182ed04" containerName="maas-api" containerID="cri-o://c681fc6902713f034c8542802660df0893e98a42fbf6333b5cd972e1d4d95877" gracePeriod=30 Jul 24 16:13:52.196584 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:52.196521 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" Jul 24 16:13:52.221159 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:52.221129 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5931b2af-399f-4b0d-ac57-3ed52182ed04-maas-api-tls\") pod \"5931b2af-399f-4b0d-ac57-3ed52182ed04\" (UID: \"5931b2af-399f-4b0d-ac57-3ed52182ed04\") " Jul 24 16:13:52.221353 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:52.221311 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptjz7\" (UniqueName: \"kubernetes.io/projected/5931b2af-399f-4b0d-ac57-3ed52182ed04-kube-api-access-ptjz7\") pod \"5931b2af-399f-4b0d-ac57-3ed52182ed04\" (UID: \"5931b2af-399f-4b0d-ac57-3ed52182ed04\") " Jul 24 16:13:52.221435 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:52.221352 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5931b2af-399f-4b0d-ac57-3ed52182ed04-openshift-service-ca\") pod \"5931b2af-399f-4b0d-ac57-3ed52182ed04\" (UID: \"5931b2af-399f-4b0d-ac57-3ed52182ed04\") " Jul 24 16:13:52.221960 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:52.221929 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5931b2af-399f-4b0d-ac57-3ed52182ed04-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5931b2af-399f-4b0d-ac57-3ed52182ed04" (UID: "5931b2af-399f-4b0d-ac57-3ed52182ed04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:13:52.223700 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:52.223673 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5931b2af-399f-4b0d-ac57-3ed52182ed04-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5931b2af-399f-4b0d-ac57-3ed52182ed04" (UID: "5931b2af-399f-4b0d-ac57-3ed52182ed04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:13:52.223970 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:52.223945 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5931b2af-399f-4b0d-ac57-3ed52182ed04-kube-api-access-ptjz7" (OuterVolumeSpecName: "kube-api-access-ptjz7") pod "5931b2af-399f-4b0d-ac57-3ed52182ed04" (UID: "5931b2af-399f-4b0d-ac57-3ed52182ed04"). InnerVolumeSpecName "kube-api-access-ptjz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:13:52.323150 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:52.323121 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptjz7\" (UniqueName: \"kubernetes.io/projected/5931b2af-399f-4b0d-ac57-3ed52182ed04-kube-api-access-ptjz7\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:13:52.323150 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:52.323146 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5931b2af-399f-4b0d-ac57-3ed52182ed04-openshift-service-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:13:52.323150 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:52.323157 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5931b2af-399f-4b0d-ac57-3ed52182ed04-maas-api-tls\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:13:53.040430 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:53.040392 2569 generic.go:356] "Generic (PLEG): container finished" podID="5931b2af-399f-4b0d-ac57-3ed52182ed04" containerID="c681fc6902713f034c8542802660df0893e98a42fbf6333b5cd972e1d4d95877" exitCode=0 Jul 24 16:13:53.040905 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:53.040442 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" Jul 24 16:13:53.040905 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:53.040463 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" event={"ID":"5931b2af-399f-4b0d-ac57-3ed52182ed04","Type":"ContainerDied","Data":"c681fc6902713f034c8542802660df0893e98a42fbf6333b5cd972e1d4d95877"} Jul 24 16:13:53.040905 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:53.040516 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t" event={"ID":"5931b2af-399f-4b0d-ac57-3ed52182ed04","Type":"ContainerDied","Data":"61cba0771de21eabfb7981a1fe245a0d6b601cdfdc7e66988d5bce5bcf8c59e4"} Jul 24 16:13:53.040905 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:53.040544 2569 scope.go:117] "RemoveContainer" containerID="c681fc6902713f034c8542802660df0893e98a42fbf6333b5cd972e1d4d95877" Jul 24 16:13:53.051536 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:53.051517 2569 scope.go:117] "RemoveContainer" containerID="c681fc6902713f034c8542802660df0893e98a42fbf6333b5cd972e1d4d95877" Jul 24 16:13:53.051777 ip-10-0-141-58 kubenswrapper[2569]: E0724 16:13:53.051756 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c681fc6902713f034c8542802660df0893e98a42fbf6333b5cd972e1d4d95877\": container with ID starting with c681fc6902713f034c8542802660df0893e98a42fbf6333b5cd972e1d4d95877 not found: ID does not exist" containerID="c681fc6902713f034c8542802660df0893e98a42fbf6333b5cd972e1d4d95877" Jul 24 16:13:53.051823 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:53.051785 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c681fc6902713f034c8542802660df0893e98a42fbf6333b5cd972e1d4d95877"} err="failed to get container status \"c681fc6902713f034c8542802660df0893e98a42fbf6333b5cd972e1d4d95877\": rpc error: code = NotFound desc = could not find container \"c681fc6902713f034c8542802660df0893e98a42fbf6333b5cd972e1d4d95877\": container with ID starting with c681fc6902713f034c8542802660df0893e98a42fbf6333b5cd972e1d4d95877 not found: ID does not exist" Jul 24 16:13:53.072433 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:53.072405 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t"] Jul 24 16:13:53.079178 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:53.079150 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c3420287-7dd96b8c74-vrd5t"] Jul 24 16:13:53.638414 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:13:53.638377 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5931b2af-399f-4b0d-ac57-3ed52182ed04" path="/var/lib/kubelet/pods/5931b2af-399f-4b0d-ac57-3ed52182ed04/volumes" Jul 24 16:15:25.659540 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:15:25.659511 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 16:15:25.671960 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:15:25.671933 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 16:15:25.672502 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:15:25.672484 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 16:15:25.677666 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:15:25.677645 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 16:20:25.714451 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:20:25.714375 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 16:20:25.714987 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:20:25.714451 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 16:20:25.720280 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:20:25.720257 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 16:20:25.720441 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:20:25.720260 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 16:21:39.612319 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.612227 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n"] Jul 24 16:21:39.612787 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.612732 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5931b2af-399f-4b0d-ac57-3ed52182ed04" containerName="maas-api" Jul 24 16:21:39.612787 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.612744 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5931b2af-399f-4b0d-ac57-3ed52182ed04" containerName="maas-api" Jul 24 16:21:39.612863 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.612813 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5931b2af-399f-4b0d-ac57-3ed52182ed04" containerName="maas-api" Jul 24 16:21:39.616042 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.616024 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n" Jul 24 16:21:39.618623 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.618598 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-573c9b\"" Jul 24 16:21:39.618794 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.618776 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-573c9b-dockercfg-h7t8m\"" Jul 24 16:21:39.633985 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.633953 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n"] Jul 24 16:21:39.634167 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.634054 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5"] Jul 24 16:21:39.637863 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.637839 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5" Jul 24 16:21:39.646115 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.646072 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5"] Jul 24 16:21:39.703800 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.703760 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4a95ff-67aa-43bd-8eff-4f089d379f7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n\" (UID: \"7c4a95ff-67aa-43bd-8eff-4f089d379f7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n" Jul 24 16:21:39.704015 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.703826 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87444c3-231b-4007-868b-e69ad3c0deac-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5\" (UID: \"f87444c3-231b-4007-868b-e69ad3c0deac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5" Jul 24 16:21:39.704091 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.704031 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4qkq\" (UniqueName: \"kubernetes.io/projected/7c4a95ff-67aa-43bd-8eff-4f089d379f7d-kube-api-access-z4qkq\") pod \"payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n\" (UID: \"7c4a95ff-67aa-43bd-8eff-4f089d379f7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n" Jul 24 16:21:39.704153 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.704099 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54ffn\" (UniqueName: \"kubernetes.io/projected/f87444c3-231b-4007-868b-e69ad3c0deac-kube-api-access-54ffn\") pod \"payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5\" (UID: \"f87444c3-231b-4007-868b-e69ad3c0deac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5" Jul 24 16:21:39.804717 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.804683 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87444c3-231b-4007-868b-e69ad3c0deac-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5\" (UID: \"f87444c3-231b-4007-868b-e69ad3c0deac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5" Jul 24 16:21:39.804913 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.804763 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4qkq\" (UniqueName: \"kubernetes.io/projected/7c4a95ff-67aa-43bd-8eff-4f089d379f7d-kube-api-access-z4qkq\") pod \"payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n\" (UID: \"7c4a95ff-67aa-43bd-8eff-4f089d379f7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n" Jul 24 16:21:39.804913 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.804791 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54ffn\" (UniqueName: \"kubernetes.io/projected/f87444c3-231b-4007-868b-e69ad3c0deac-kube-api-access-54ffn\") pod \"payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5\" (UID: \"f87444c3-231b-4007-868b-e69ad3c0deac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5" Jul 24 16:21:39.804913 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.804822 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4a95ff-67aa-43bd-8eff-4f089d379f7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n\" (UID: \"7c4a95ff-67aa-43bd-8eff-4f089d379f7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n" Jul 24 16:21:39.805551 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.805530 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4a95ff-67aa-43bd-8eff-4f089d379f7d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n\" (UID: \"7c4a95ff-67aa-43bd-8eff-4f089d379f7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n" Jul 24 16:21:39.805551 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.805541 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87444c3-231b-4007-868b-e69ad3c0deac-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5\" (UID: \"f87444c3-231b-4007-868b-e69ad3c0deac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5" Jul 24 16:21:39.814096 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.814070 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4qkq\" (UniqueName: \"kubernetes.io/projected/7c4a95ff-67aa-43bd-8eff-4f089d379f7d-kube-api-access-z4qkq\") pod \"payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n\" (UID: \"7c4a95ff-67aa-43bd-8eff-4f089d379f7d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n" Jul 24 16:21:39.815179 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.815161 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54ffn\" (UniqueName: \"kubernetes.io/projected/f87444c3-231b-4007-868b-e69ad3c0deac-kube-api-access-54ffn\") pod \"payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5\" (UID: \"f87444c3-231b-4007-868b-e69ad3c0deac\") " pod="openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5" Jul 24 16:21:39.929772 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.929729 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n" Jul 24 16:21:39.951180 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:39.951146 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5" Jul 24 16:21:40.100180 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:40.100147 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n"] Jul 24 16:21:40.102045 ip-10-0-141-58 kubenswrapper[2569]: W0724 16:21:40.102009 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4a95ff_67aa_43bd_8eff_4f089d379f7d.slice/crio-10a674b8282041169fb73b54ce33ec5f8eff24c9c7ea39b31c2fc686d17f02ec WatchSource:0}: Error finding container 10a674b8282041169fb73b54ce33ec5f8eff24c9c7ea39b31c2fc686d17f02ec: Status 404 returned error can't find the container with id 10a674b8282041169fb73b54ce33ec5f8eff24c9c7ea39b31c2fc686d17f02ec Jul 24 16:21:40.136498 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:40.136473 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5"] Jul 24 16:21:40.148952 ip-10-0-141-58 kubenswrapper[2569]: W0724 16:21:40.148928 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87444c3_231b_4007_868b_e69ad3c0deac.slice/crio-02f0cd43b85189fb2c0feedca737f58dc577e04483b85ea0c4d677158b8f0b24 WatchSource:0}: Error finding container 02f0cd43b85189fb2c0feedca737f58dc577e04483b85ea0c4d677158b8f0b24: Status 404 returned error can't find the container with id 02f0cd43b85189fb2c0feedca737f58dc577e04483b85ea0c4d677158b8f0b24 Jul 24 16:21:41.024641 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:41.024608 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5" event={"ID":"f87444c3-231b-4007-868b-e69ad3c0deac","Type":"ContainerStarted","Data":"070f4c5339ded5d2888bf29581ddc7c469d84a2c0cb07de21e1349a557be824b"} Jul 24 16:21:41.025116 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:41.024765 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5" Jul 24 16:21:41.025116 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:41.024788 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5" event={"ID":"f87444c3-231b-4007-868b-e69ad3c0deac","Type":"ContainerStarted","Data":"02f0cd43b85189fb2c0feedca737f58dc577e04483b85ea0c4d677158b8f0b24"} Jul 24 16:21:41.026638 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:41.026611 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n" event={"ID":"7c4a95ff-67aa-43bd-8eff-4f089d379f7d","Type":"ContainerStarted","Data":"bab657d567e04b065985adf76159525683c12788f46e21633b1c2b17f5d9755c"} Jul 24 16:21:41.026839 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:41.026824 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n" event={"ID":"7c4a95ff-67aa-43bd-8eff-4f089d379f7d","Type":"ContainerStarted","Data":"10a674b8282041169fb73b54ce33ec5f8eff24c9c7ea39b31c2fc686d17f02ec"} Jul 24 16:21:41.026947 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:41.026935 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n" Jul 24 16:21:41.043370 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:41.043323 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5" podStartSLOduration=2.043306564 podStartE2EDuration="2.043306564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:21:41.042025736 +0000 UTC m=+2175.959996037" watchObservedRunningTime="2026-07-24 16:21:41.043306564 +0000 UTC m=+2175.961276869" Jul 24 16:21:41.060442 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:41.060383 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n" podStartSLOduration=2.060364029 podStartE2EDuration="2.060364029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:21:41.058794665 +0000 UTC m=+2175.976764965" watchObservedRunningTime="2026-07-24 16:21:41.060364029 +0000 UTC m=+2175.978334329" Jul 24 16:21:52.038716 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:52.038665 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n" podUID="7c4a95ff-67aa-43bd-8eff-4f089d379f7d" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.71:9004: connect: connection refused" Jul 24 16:21:52.039293 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:21:52.039154 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5" Jul 24 16:22:02.037067 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:22:02.037037 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n" Jul 24 16:24:07.997478 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:07.997441 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5"] Jul 24 16:24:07.998064 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:07.997702 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5" podUID="f87444c3-231b-4007-868b-e69ad3c0deac" containerName="payload-processing" containerID="cri-o://070f4c5339ded5d2888bf29581ddc7c469d84a2c0cb07de21e1349a557be824b" gracePeriod=30 Jul 24 16:24:08.005727 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.005696 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n"] Jul 24 16:24:08.006007 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.005949 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n" podUID="7c4a95ff-67aa-43bd-8eff-4f089d379f7d" containerName="payload-pre-processing" containerID="cri-o://bab657d567e04b065985adf76159525683c12788f46e21633b1c2b17f5d9755c" gracePeriod=30 Jul 24 16:24:08.264202 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.264123 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5" Jul 24 16:24:08.293362 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.293332 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n" Jul 24 16:24:08.365636 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.365601 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87444c3-231b-4007-868b-e69ad3c0deac-plugins-config-volume\") pod \"f87444c3-231b-4007-868b-e69ad3c0deac\" (UID: \"f87444c3-231b-4007-868b-e69ad3c0deac\") " Jul 24 16:24:08.365827 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.365685 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4qkq\" (UniqueName: \"kubernetes.io/projected/7c4a95ff-67aa-43bd-8eff-4f089d379f7d-kube-api-access-z4qkq\") pod \"7c4a95ff-67aa-43bd-8eff-4f089d379f7d\" (UID: \"7c4a95ff-67aa-43bd-8eff-4f089d379f7d\") " Jul 24 16:24:08.365827 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.365764 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4a95ff-67aa-43bd-8eff-4f089d379f7d-plugins-config-volume\") pod \"7c4a95ff-67aa-43bd-8eff-4f089d379f7d\" (UID: \"7c4a95ff-67aa-43bd-8eff-4f089d379f7d\") " Jul 24 16:24:08.365827 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.365800 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54ffn\" (UniqueName: \"kubernetes.io/projected/f87444c3-231b-4007-868b-e69ad3c0deac-kube-api-access-54ffn\") pod \"f87444c3-231b-4007-868b-e69ad3c0deac\" (UID: \"f87444c3-231b-4007-868b-e69ad3c0deac\") " Jul 24 16:24:08.366063 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.366036 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87444c3-231b-4007-868b-e69ad3c0deac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f87444c3-231b-4007-868b-e69ad3c0deac" (UID: "f87444c3-231b-4007-868b-e69ad3c0deac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:24:08.366162 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.366135 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c4a95ff-67aa-43bd-8eff-4f089d379f7d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c4a95ff-67aa-43bd-8eff-4f089d379f7d" (UID: "7c4a95ff-67aa-43bd-8eff-4f089d379f7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:24:08.367807 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.367784 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4a95ff-67aa-43bd-8eff-4f089d379f7d-kube-api-access-z4qkq" (OuterVolumeSpecName: "kube-api-access-z4qkq") pod "7c4a95ff-67aa-43bd-8eff-4f089d379f7d" (UID: "7c4a95ff-67aa-43bd-8eff-4f089d379f7d"). InnerVolumeSpecName "kube-api-access-z4qkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:24:08.367905 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.367889 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f87444c3-231b-4007-868b-e69ad3c0deac-kube-api-access-54ffn" (OuterVolumeSpecName: "kube-api-access-54ffn") pod "f87444c3-231b-4007-868b-e69ad3c0deac" (UID: "f87444c3-231b-4007-868b-e69ad3c0deac"). InnerVolumeSpecName "kube-api-access-54ffn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:24:08.466734 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.466698 2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4a95ff-67aa-43bd-8eff-4f089d379f7d-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:24:08.466734 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.466728 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54ffn\" (UniqueName: \"kubernetes.io/projected/f87444c3-231b-4007-868b-e69ad3c0deac-kube-api-access-54ffn\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:24:08.466734 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.466738 2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f87444c3-231b-4007-868b-e69ad3c0deac-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:24:08.466969 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.466747 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4qkq\" (UniqueName: \"kubernetes.io/projected/7c4a95ff-67aa-43bd-8eff-4f089d379f7d-kube-api-access-z4qkq\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:24:08.965399 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.965367 2569 generic.go:356] "Generic (PLEG): container finished" podID="7c4a95ff-67aa-43bd-8eff-4f089d379f7d" containerID="bab657d567e04b065985adf76159525683c12788f46e21633b1c2b17f5d9755c" exitCode=0 Jul 24 16:24:08.965591 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.965408 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n" Jul 24 16:24:08.965591 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.965456 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n" event={"ID":"7c4a95ff-67aa-43bd-8eff-4f089d379f7d","Type":"ContainerDied","Data":"bab657d567e04b065985adf76159525683c12788f46e21633b1c2b17f5d9755c"} Jul 24 16:24:08.965591 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.965510 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n" event={"ID":"7c4a95ff-67aa-43bd-8eff-4f089d379f7d","Type":"ContainerDied","Data":"10a674b8282041169fb73b54ce33ec5f8eff24c9c7ea39b31c2fc686d17f02ec"} Jul 24 16:24:08.965591 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.965549 2569 scope.go:117] "RemoveContainer" containerID="bab657d567e04b065985adf76159525683c12788f46e21633b1c2b17f5d9755c" Jul 24 16:24:08.966878 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.966849 2569 generic.go:356] "Generic (PLEG): container finished" podID="f87444c3-231b-4007-868b-e69ad3c0deac" containerID="070f4c5339ded5d2888bf29581ddc7c469d84a2c0cb07de21e1349a557be824b" exitCode=0 Jul 24 16:24:08.966965 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.966895 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5" Jul 24 16:24:08.966965 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.966893 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5" event={"ID":"f87444c3-231b-4007-868b-e69ad3c0deac","Type":"ContainerDied","Data":"070f4c5339ded5d2888bf29581ddc7c469d84a2c0cb07de21e1349a557be824b"} Jul 24 16:24:08.967067 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.967000 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5" event={"ID":"f87444c3-231b-4007-868b-e69ad3c0deac","Type":"ContainerDied","Data":"02f0cd43b85189fb2c0feedca737f58dc577e04483b85ea0c4d677158b8f0b24"} Jul 24 16:24:08.977124 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.977104 2569 scope.go:117] "RemoveContainer" containerID="bab657d567e04b065985adf76159525683c12788f46e21633b1c2b17f5d9755c" Jul 24 16:24:08.977472 ip-10-0-141-58 kubenswrapper[2569]: E0724 16:24:08.977445 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab657d567e04b065985adf76159525683c12788f46e21633b1c2b17f5d9755c\": container with ID starting with bab657d567e04b065985adf76159525683c12788f46e21633b1c2b17f5d9755c not found: ID does not exist" containerID="bab657d567e04b065985adf76159525683c12788f46e21633b1c2b17f5d9755c" Jul 24 16:24:08.977574 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.977481 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab657d567e04b065985adf76159525683c12788f46e21633b1c2b17f5d9755c"} err="failed to get container status \"bab657d567e04b065985adf76159525683c12788f46e21633b1c2b17f5d9755c\": rpc error: code = NotFound desc = could not find container \"bab657d567e04b065985adf76159525683c12788f46e21633b1c2b17f5d9755c\": container with ID starting with bab657d567e04b065985adf76159525683c12788f46e21633b1c2b17f5d9755c not found: ID does not exist" Jul 24 16:24:08.977574 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.977505 2569 scope.go:117] "RemoveContainer" containerID="070f4c5339ded5d2888bf29581ddc7c469d84a2c0cb07de21e1349a557be824b" Jul 24 16:24:08.986807 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.986789 2569 scope.go:117] "RemoveContainer" containerID="070f4c5339ded5d2888bf29581ddc7c469d84a2c0cb07de21e1349a557be824b" Jul 24 16:24:08.987075 ip-10-0-141-58 kubenswrapper[2569]: E0724 16:24:08.987052 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"070f4c5339ded5d2888bf29581ddc7c469d84a2c0cb07de21e1349a557be824b\": container with ID starting with 070f4c5339ded5d2888bf29581ddc7c469d84a2c0cb07de21e1349a557be824b not found: ID does not exist" containerID="070f4c5339ded5d2888bf29581ddc7c469d84a2c0cb07de21e1349a557be824b" Jul 24 16:24:08.987141 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.987084 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"070f4c5339ded5d2888bf29581ddc7c469d84a2c0cb07de21e1349a557be824b"} err="failed to get container status \"070f4c5339ded5d2888bf29581ddc7c469d84a2c0cb07de21e1349a557be824b\": rpc error: code = NotFound desc = could not find container \"070f4c5339ded5d2888bf29581ddc7c469d84a2c0cb07de21e1349a557be824b\": container with ID starting with 070f4c5339ded5d2888bf29581ddc7c469d84a2c0cb07de21e1349a557be824b not found: ID does not exist" Jul 24 16:24:08.992503 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.992482 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n"] Jul 24 16:24:08.996657 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:08.996636 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-573c9b-664b6f58d4-h444n"] Jul 24 16:24:09.007552 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:09.007531 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5"] Jul 24 16:24:09.011628 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:09.011605 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-573c9b-9f86b6958-vddw5"] Jul 24 16:24:09.628884 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:09.628853 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c4a95ff-67aa-43bd-8eff-4f089d379f7d" path="/var/lib/kubelet/pods/7c4a95ff-67aa-43bd-8eff-4f089d379f7d/volumes" Jul 24 16:24:09.629247 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:09.629231 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f87444c3-231b-4007-868b-e69ad3c0deac" path="/var/lib/kubelet/pods/f87444c3-231b-4007-868b-e69ad3c0deac/volumes" Jul 24 16:24:32.659869 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.659833 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8"] Jul 24 16:24:32.660353 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.660338 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f87444c3-231b-4007-868b-e69ad3c0deac" containerName="payload-processing" Jul 24 16:24:32.660353 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.660355 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f87444c3-231b-4007-868b-e69ad3c0deac" containerName="payload-processing" Jul 24 16:24:32.660443 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.660378 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c4a95ff-67aa-43bd-8eff-4f089d379f7d" containerName="payload-pre-processing" Jul 24 16:24:32.660443 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.660385 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c4a95ff-67aa-43bd-8eff-4f089d379f7d" containerName="payload-pre-processing" Jul 24 16:24:32.660511 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.660468 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c4a95ff-67aa-43bd-8eff-4f089d379f7d" containerName="payload-pre-processing" Jul 24 16:24:32.660511 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.660480 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f87444c3-231b-4007-868b-e69ad3c0deac" containerName="payload-processing" Jul 24 16:24:32.663718 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.663700 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8" Jul 24 16:24:32.669734 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.669716 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-966c3e-dockercfg-lrk4h\"" Jul 24 16:24:32.670630 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.670306 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-966c3e\"" Jul 24 16:24:32.680983 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.680960 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8"] Jul 24 16:24:32.748773 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.748740 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8"] Jul 24 16:24:32.752446 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.752425 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8" Jul 24 16:24:32.764487 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.764461 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8"] Jul 24 16:24:32.812120 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.812088 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz24x\" (UniqueName: \"kubernetes.io/projected/519a03de-56f6-48a8-b782-2b261f42783e-kube-api-access-mz24x\") pod \"payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8\" (UID: \"519a03de-56f6-48a8-b782-2b261f42783e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8" Jul 24 16:24:32.812329 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.812144 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519a03de-56f6-48a8-b782-2b261f42783e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8\" (UID: \"519a03de-56f6-48a8-b782-2b261f42783e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8" Jul 24 16:24:32.913410 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.913314 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2qc4\" (UniqueName: \"kubernetes.io/projected/c3b571de-59dd-48f0-831e-6f26bd58c734-kube-api-access-n2qc4\") pod \"payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8\" (UID: \"c3b571de-59dd-48f0-831e-6f26bd58c734\") " pod="openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8" Jul 24 16:24:32.913410 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.913375 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz24x\" (UniqueName: \"kubernetes.io/projected/519a03de-56f6-48a8-b782-2b261f42783e-kube-api-access-mz24x\") pod \"payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8\" (UID: \"519a03de-56f6-48a8-b782-2b261f42783e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8" Jul 24 16:24:32.913660 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.913419 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b571de-59dd-48f0-831e-6f26bd58c734-plugins-config-volume\") pod \"payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8\" (UID: \"c3b571de-59dd-48f0-831e-6f26bd58c734\") " pod="openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8" Jul 24 16:24:32.913660 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.913461 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519a03de-56f6-48a8-b782-2b261f42783e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8\" (UID: \"519a03de-56f6-48a8-b782-2b261f42783e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8" Jul 24 16:24:32.914166 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.914141 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519a03de-56f6-48a8-b782-2b261f42783e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8\" (UID: \"519a03de-56f6-48a8-b782-2b261f42783e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8" Jul 24 16:24:32.929993 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.929956 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz24x\" (UniqueName: \"kubernetes.io/projected/519a03de-56f6-48a8-b782-2b261f42783e-kube-api-access-mz24x\") pod \"payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8\" (UID: \"519a03de-56f6-48a8-b782-2b261f42783e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8" Jul 24 16:24:32.974670 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:32.974637 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8" Jul 24 16:24:33.014664 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:33.014628 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b571de-59dd-48f0-831e-6f26bd58c734-plugins-config-volume\") pod \"payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8\" (UID: \"c3b571de-59dd-48f0-831e-6f26bd58c734\") " pod="openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8" Jul 24 16:24:33.014813 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:33.014731 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2qc4\" (UniqueName: \"kubernetes.io/projected/c3b571de-59dd-48f0-831e-6f26bd58c734-kube-api-access-n2qc4\") pod \"payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8\" (UID: \"c3b571de-59dd-48f0-831e-6f26bd58c734\") " pod="openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8" Jul 24 16:24:33.015641 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:33.015615 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b571de-59dd-48f0-831e-6f26bd58c734-plugins-config-volume\") pod \"payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8\" (UID: \"c3b571de-59dd-48f0-831e-6f26bd58c734\") " pod="openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8" Jul 24 16:24:33.030632 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:33.030441 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2qc4\" (UniqueName: \"kubernetes.io/projected/c3b571de-59dd-48f0-831e-6f26bd58c734-kube-api-access-n2qc4\") pod \"payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8\" (UID: \"c3b571de-59dd-48f0-831e-6f26bd58c734\") " pod="openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8" Jul 24 16:24:33.064494 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:33.064453 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8" Jul 24 16:24:33.243192 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:33.243159 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8"] Jul 24 16:24:33.244564 ip-10-0-141-58 kubenswrapper[2569]: W0724 16:24:33.244541 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b571de_59dd_48f0_831e_6f26bd58c734.slice/crio-84cfb372d9bf5b9120db3fac90cbf77758b060a53f7b4a68717d89b4dca1b4a3 WatchSource:0}: Error finding container 84cfb372d9bf5b9120db3fac90cbf77758b060a53f7b4a68717d89b4dca1b4a3: Status 404 returned error can't find the container with id 84cfb372d9bf5b9120db3fac90cbf77758b060a53f7b4a68717d89b4dca1b4a3 Jul 24 16:24:33.354744 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:33.354715 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8"] Jul 24 16:24:33.357105 ip-10-0-141-58 kubenswrapper[2569]: W0724 16:24:33.357076 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519a03de_56f6_48a8_b782_2b261f42783e.slice/crio-3b5d9bf870ec472485b7d5f8e4eab6c4b8dc13bfeb3acabed31e00841778a361 WatchSource:0}: Error finding container 3b5d9bf870ec472485b7d5f8e4eab6c4b8dc13bfeb3acabed31e00841778a361: Status 404 returned error can't find the container with id 3b5d9bf870ec472485b7d5f8e4eab6c4b8dc13bfeb3acabed31e00841778a361 Jul 24 16:24:34.125857 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:34.125721 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8" event={"ID":"c3b571de-59dd-48f0-831e-6f26bd58c734","Type":"ContainerStarted","Data":"f649a0170fa2a7a4bc21e5070960ad014687d533364f3ef9cddd06c1e5c53cd8"} Jul 24 16:24:34.125857 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:34.125802 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8" Jul 24 16:24:34.125857 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:34.125825 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8" event={"ID":"c3b571de-59dd-48f0-831e-6f26bd58c734","Type":"ContainerStarted","Data":"84cfb372d9bf5b9120db3fac90cbf77758b060a53f7b4a68717d89b4dca1b4a3"} Jul 24 16:24:34.127633 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:34.127597 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8" event={"ID":"519a03de-56f6-48a8-b782-2b261f42783e","Type":"ContainerStarted","Data":"a029a72636a7b43ab8570b040c4d7b35e8678e1618fbbaad44b40009214993ea"} Jul 24 16:24:34.127781 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:34.127653 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8" event={"ID":"519a03de-56f6-48a8-b782-2b261f42783e","Type":"ContainerStarted","Data":"3b5d9bf870ec472485b7d5f8e4eab6c4b8dc13bfeb3acabed31e00841778a361"} Jul 24 16:24:34.128142 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:34.128123 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8" Jul 24 16:24:34.147504 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:34.147449 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8" podStartSLOduration=2.147433665 podStartE2EDuration="2.147433665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:24:34.144189562 +0000 UTC m=+2349.062159862" watchObservedRunningTime="2026-07-24 16:24:34.147433665 +0000 UTC m=+2349.065403966" Jul 24 16:24:34.169864 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:34.169806 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8" podStartSLOduration=2.169791491 podStartE2EDuration="2.169791491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:24:34.168686576 +0000 UTC m=+2349.086656878" watchObservedRunningTime="2026-07-24 16:24:34.169791491 +0000 UTC m=+2349.087761790" Jul 24 16:24:45.137683 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:45.137653 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8" Jul 24 16:24:46.144227 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:24:46.144189 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8" Jul 24 16:25:25.762073 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:25.762041 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 16:25:25.763520 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:25.763499 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 16:25:25.768456 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:25.768432 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 16:25:25.769661 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:25.769639 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 16:25:27.577080 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:27.577045 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8"] Jul 24 16:25:27.577561 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:27.577316 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8" podUID="c3b571de-59dd-48f0-831e-6f26bd58c734" containerName="payload-processing" containerID="cri-o://f649a0170fa2a7a4bc21e5070960ad014687d533364f3ef9cddd06c1e5c53cd8" gracePeriod=30 Jul 24 16:25:27.599715 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:27.599686 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8"] Jul 24 16:25:27.599980 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:27.599922 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8" podUID="519a03de-56f6-48a8-b782-2b261f42783e" containerName="payload-pre-processing" containerID="cri-o://a029a72636a7b43ab8570b040c4d7b35e8678e1618fbbaad44b40009214993ea" gracePeriod=30 Jul 24 16:25:27.887674 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:27.887650 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8" Jul 24 16:25:27.890627 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:27.890607 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8" Jul 24 16:25:27.955333 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:27.955302 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b571de-59dd-48f0-831e-6f26bd58c734-plugins-config-volume\") pod \"c3b571de-59dd-48f0-831e-6f26bd58c734\" (UID: \"c3b571de-59dd-48f0-831e-6f26bd58c734\") " Jul 24 16:25:27.955525 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:27.955396 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2qc4\" (UniqueName: \"kubernetes.io/projected/c3b571de-59dd-48f0-831e-6f26bd58c734-kube-api-access-n2qc4\") pod \"c3b571de-59dd-48f0-831e-6f26bd58c734\" (UID: \"c3b571de-59dd-48f0-831e-6f26bd58c734\") " Jul 24 16:25:27.955525 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:27.955420 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519a03de-56f6-48a8-b782-2b261f42783e-plugins-config-volume\") pod \"519a03de-56f6-48a8-b782-2b261f42783e\" (UID: \"519a03de-56f6-48a8-b782-2b261f42783e\") " Jul 24 16:25:27.955525 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:27.955440 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz24x\" (UniqueName: \"kubernetes.io/projected/519a03de-56f6-48a8-b782-2b261f42783e-kube-api-access-mz24x\") pod \"519a03de-56f6-48a8-b782-2b261f42783e\" (UID: \"519a03de-56f6-48a8-b782-2b261f42783e\") " Jul 24 16:25:27.955849 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:27.955820 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b571de-59dd-48f0-831e-6f26bd58c734-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3b571de-59dd-48f0-831e-6f26bd58c734" (UID: "c3b571de-59dd-48f0-831e-6f26bd58c734"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:25:27.955849 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:27.955844 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519a03de-56f6-48a8-b782-2b261f42783e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "519a03de-56f6-48a8-b782-2b261f42783e" (UID: "519a03de-56f6-48a8-b782-2b261f42783e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:25:27.957675 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:27.957652 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b571de-59dd-48f0-831e-6f26bd58c734-kube-api-access-n2qc4" (OuterVolumeSpecName: "kube-api-access-n2qc4") pod "c3b571de-59dd-48f0-831e-6f26bd58c734" (UID: "c3b571de-59dd-48f0-831e-6f26bd58c734"). InnerVolumeSpecName "kube-api-access-n2qc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:25:27.957771 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:27.957681 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519a03de-56f6-48a8-b782-2b261f42783e-kube-api-access-mz24x" (OuterVolumeSpecName: "kube-api-access-mz24x") pod "519a03de-56f6-48a8-b782-2b261f42783e" (UID: "519a03de-56f6-48a8-b782-2b261f42783e"). InnerVolumeSpecName "kube-api-access-mz24x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:25:28.056684 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.056641 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2qc4\" (UniqueName: \"kubernetes.io/projected/c3b571de-59dd-48f0-831e-6f26bd58c734-kube-api-access-n2qc4\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:25:28.056684 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.056675 2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/519a03de-56f6-48a8-b782-2b261f42783e-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:25:28.056684 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.056684 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz24x\" (UniqueName: \"kubernetes.io/projected/519a03de-56f6-48a8-b782-2b261f42783e-kube-api-access-mz24x\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:25:28.056684 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.056693 2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b571de-59dd-48f0-831e-6f26bd58c734-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:25:28.463776 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.463741 2569 generic.go:356] "Generic (PLEG): container finished" podID="c3b571de-59dd-48f0-831e-6f26bd58c734" containerID="f649a0170fa2a7a4bc21e5070960ad014687d533364f3ef9cddd06c1e5c53cd8" exitCode=0 Jul 24 16:25:28.463961 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.463792 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8" Jul 24 16:25:28.463961 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.463829 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8" event={"ID":"c3b571de-59dd-48f0-831e-6f26bd58c734","Type":"ContainerDied","Data":"f649a0170fa2a7a4bc21e5070960ad014687d533364f3ef9cddd06c1e5c53cd8"} Jul 24 16:25:28.463961 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.463882 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8" event={"ID":"c3b571de-59dd-48f0-831e-6f26bd58c734","Type":"ContainerDied","Data":"84cfb372d9bf5b9120db3fac90cbf77758b060a53f7b4a68717d89b4dca1b4a3"} Jul 24 16:25:28.463961 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.463912 2569 scope.go:117] "RemoveContainer" containerID="f649a0170fa2a7a4bc21e5070960ad014687d533364f3ef9cddd06c1e5c53cd8" Jul 24 16:25:28.465166 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.465143 2569 generic.go:356] "Generic (PLEG): container finished" podID="519a03de-56f6-48a8-b782-2b261f42783e" containerID="a029a72636a7b43ab8570b040c4d7b35e8678e1618fbbaad44b40009214993ea" exitCode=0 Jul 24 16:25:28.465275 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.465171 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8" event={"ID":"519a03de-56f6-48a8-b782-2b261f42783e","Type":"ContainerDied","Data":"a029a72636a7b43ab8570b040c4d7b35e8678e1618fbbaad44b40009214993ea"} Jul 24 16:25:28.465275 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.465184 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8" Jul 24 16:25:28.465275 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.465228 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8" event={"ID":"519a03de-56f6-48a8-b782-2b261f42783e","Type":"ContainerDied","Data":"3b5d9bf870ec472485b7d5f8e4eab6c4b8dc13bfeb3acabed31e00841778a361"} Jul 24 16:25:28.483180 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.483163 2569 scope.go:117] "RemoveContainer" containerID="f649a0170fa2a7a4bc21e5070960ad014687d533364f3ef9cddd06c1e5c53cd8" Jul 24 16:25:28.483512 ip-10-0-141-58 kubenswrapper[2569]: E0724 16:25:28.483481 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f649a0170fa2a7a4bc21e5070960ad014687d533364f3ef9cddd06c1e5c53cd8\": container with ID starting with f649a0170fa2a7a4bc21e5070960ad014687d533364f3ef9cddd06c1e5c53cd8 not found: ID does not exist" containerID="f649a0170fa2a7a4bc21e5070960ad014687d533364f3ef9cddd06c1e5c53cd8" Jul 24 16:25:28.483612 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.483519 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f649a0170fa2a7a4bc21e5070960ad014687d533364f3ef9cddd06c1e5c53cd8"} err="failed to get container status \"f649a0170fa2a7a4bc21e5070960ad014687d533364f3ef9cddd06c1e5c53cd8\": rpc error: code = NotFound desc = could not find container \"f649a0170fa2a7a4bc21e5070960ad014687d533364f3ef9cddd06c1e5c53cd8\": container with ID starting with f649a0170fa2a7a4bc21e5070960ad014687d533364f3ef9cddd06c1e5c53cd8 not found: ID does not exist" Jul 24 16:25:28.483612 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.483537 2569 scope.go:117] "RemoveContainer" containerID="a029a72636a7b43ab8570b040c4d7b35e8678e1618fbbaad44b40009214993ea" Jul 24 16:25:28.492017 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.491999 2569 scope.go:117] "RemoveContainer" containerID="a029a72636a7b43ab8570b040c4d7b35e8678e1618fbbaad44b40009214993ea" Jul 24 16:25:28.492447 ip-10-0-141-58 kubenswrapper[2569]: E0724 16:25:28.492359 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a029a72636a7b43ab8570b040c4d7b35e8678e1618fbbaad44b40009214993ea\": container with ID starting with a029a72636a7b43ab8570b040c4d7b35e8678e1618fbbaad44b40009214993ea not found: ID does not exist" containerID="a029a72636a7b43ab8570b040c4d7b35e8678e1618fbbaad44b40009214993ea" Jul 24 16:25:28.492447 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.492394 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a029a72636a7b43ab8570b040c4d7b35e8678e1618fbbaad44b40009214993ea"} err="failed to get container status \"a029a72636a7b43ab8570b040c4d7b35e8678e1618fbbaad44b40009214993ea\": rpc error: code = NotFound desc = could not find container \"a029a72636a7b43ab8570b040c4d7b35e8678e1618fbbaad44b40009214993ea\": container with ID starting with a029a72636a7b43ab8570b040c4d7b35e8678e1618fbbaad44b40009214993ea not found: ID does not exist" Jul 24 16:25:28.495497 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.495478 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8"] Jul 24 16:25:28.502708 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.502687 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-966c3e-78dd777b68-cj5j8"] Jul 24 16:25:28.517509 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.517481 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8"] Jul 24 16:25:28.519953 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:28.519934 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-966c3e-7b7669f7f8-hphr8"] Jul 24 16:25:29.629834 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:29.629801 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="519a03de-56f6-48a8-b782-2b261f42783e" path="/var/lib/kubelet/pods/519a03de-56f6-48a8-b782-2b261f42783e/volumes" Jul 24 16:25:29.630260 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:29.630243 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3b571de-59dd-48f0-831e-6f26bd58c734" path="/var/lib/kubelet/pods/c3b571de-59dd-48f0-831e-6f26bd58c734/volumes" Jul 24 16:25:51.984269 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:51.984134 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25"] Jul 24 16:25:51.984839 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:51.984664 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="519a03de-56f6-48a8-b782-2b261f42783e" containerName="payload-pre-processing" Jul 24 16:25:51.984839 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:51.984677 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="519a03de-56f6-48a8-b782-2b261f42783e" containerName="payload-pre-processing" Jul 24 16:25:51.984839 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:51.984692 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3b571de-59dd-48f0-831e-6f26bd58c734" containerName="payload-processing" Jul 24 16:25:51.984839 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:51.984697 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3b571de-59dd-48f0-831e-6f26bd58c734" containerName="payload-processing" Jul 24 16:25:51.984839 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:51.984799 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3b571de-59dd-48f0-831e-6f26bd58c734" containerName="payload-processing" Jul 24 16:25:51.984839 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:51.984811 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="519a03de-56f6-48a8-b782-2b261f42783e" containerName="payload-pre-processing" Jul 24 16:25:51.988325 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:51.988305 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25" Jul 24 16:25:51.990928 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:51.990904 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-82adcb-dockercfg-26xd7\"" Jul 24 16:25:51.991055 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:51.990916 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-82adcb\"" Jul 24 16:25:51.998359 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:51.998336 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25"] Jul 24 16:25:52.090068 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:52.090032 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnfbg\" (UniqueName: \"kubernetes.io/projected/981696de-6034-44af-b650-efa0ddfc2788-kube-api-access-lnfbg\") pod \"payload-processing-e2e-shared-b-82adcb-785755576b-dvb25\" (UID: \"981696de-6034-44af-b650-efa0ddfc2788\") " pod="openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25" Jul 24 16:25:52.090274 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:52.090111 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981696de-6034-44af-b650-efa0ddfc2788-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-82adcb-785755576b-dvb25\" (UID: \"981696de-6034-44af-b650-efa0ddfc2788\") " pod="openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25" Jul 24 16:25:52.190806 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:52.190760 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981696de-6034-44af-b650-efa0ddfc2788-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-82adcb-785755576b-dvb25\" (UID: \"981696de-6034-44af-b650-efa0ddfc2788\") " pod="openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25" Jul 24 16:25:52.190996 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:52.190900 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnfbg\" (UniqueName: \"kubernetes.io/projected/981696de-6034-44af-b650-efa0ddfc2788-kube-api-access-lnfbg\") pod \"payload-processing-e2e-shared-b-82adcb-785755576b-dvb25\" (UID: \"981696de-6034-44af-b650-efa0ddfc2788\") " pod="openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25" Jul 24 16:25:52.192061 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:52.192032 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981696de-6034-44af-b650-efa0ddfc2788-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-82adcb-785755576b-dvb25\" (UID: \"981696de-6034-44af-b650-efa0ddfc2788\") " pod="openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25" Jul 24 16:25:52.200115 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:52.200085 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnfbg\" (UniqueName: \"kubernetes.io/projected/981696de-6034-44af-b650-efa0ddfc2788-kube-api-access-lnfbg\") pod \"payload-processing-e2e-shared-b-82adcb-785755576b-dvb25\" (UID: \"981696de-6034-44af-b650-efa0ddfc2788\") " pod="openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25" Jul 24 16:25:52.301414 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:52.301329 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25" Jul 24 16:25:52.455221 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:52.455180 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25"] Jul 24 16:25:52.457018 ip-10-0-141-58 kubenswrapper[2569]: W0724 16:25:52.456994 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981696de_6034_44af_b650_efa0ddfc2788.slice/crio-3ae6d52736e8c0c20057aea7c57ade46ed20f0dd5f92b8bec7a236622caae932 WatchSource:0}: Error finding container 3ae6d52736e8c0c20057aea7c57ade46ed20f0dd5f92b8bec7a236622caae932: Status 404 returned error can't find the container with id 3ae6d52736e8c0c20057aea7c57ade46ed20f0dd5f92b8bec7a236622caae932 Jul 24 16:25:52.621364 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:52.621262 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25" event={"ID":"981696de-6034-44af-b650-efa0ddfc2788","Type":"ContainerStarted","Data":"c6a84d72d940dd08b3926e458ec3337b20e4567b513b526332a10b4a0da3e866"} Jul 24 16:25:52.621364 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:52.621317 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25" Jul 24 16:25:52.621364 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:52.621341 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25" event={"ID":"981696de-6034-44af-b650-efa0ddfc2788","Type":"ContainerStarted","Data":"3ae6d52736e8c0c20057aea7c57ade46ed20f0dd5f92b8bec7a236622caae932"} Jul 24 16:25:52.657743 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:25:52.657683 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25" podStartSLOduration=1.657665981 podStartE2EDuration="1.657665981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:25:52.655066819 +0000 UTC m=+2427.573037120" watchObservedRunningTime="2026-07-24 16:25:52.657665981 +0000 UTC m=+2427.575636298" Jul 24 16:26:03.629628 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:26:03.629597 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25" Jul 24 16:30:25.814429 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:30:25.814341 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 16:30:25.817549 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:30:25.817522 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 16:30:25.820056 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:30:25.820034 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 16:30:25.823049 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:30:25.823028 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 16:31:58.169065 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:31:58.168986 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7462-139787518108336"] Jul 24 16:31:58.172538 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:31:58.172521 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518108336" Jul 24 16:31:58.175390 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:31:58.175369 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7zspr\"" Jul 24 16:31:58.183392 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:31:58.183368 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-139787518108336"] Jul 24 16:31:58.294926 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:31:58.294889 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm749\" (UniqueName: \"kubernetes.io/projected/d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7-kube-api-access-zm749\") pod \"test-curl-7462-139787518108336\" (UID: \"d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7\") " pod="opendatahub/test-curl-7462-139787518108336" Jul 24 16:31:58.395957 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:31:58.395915 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm749\" (UniqueName: \"kubernetes.io/projected/d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7-kube-api-access-zm749\") pod \"test-curl-7462-139787518108336\" (UID: \"d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7\") " pod="opendatahub/test-curl-7462-139787518108336" Jul 24 16:31:58.405832 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:31:58.405800 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm749\" (UniqueName: \"kubernetes.io/projected/d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7-kube-api-access-zm749\") pod \"test-curl-7462-139787518108336\" (UID: \"d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7\") " pod="opendatahub/test-curl-7462-139787518108336" Jul 24 16:31:58.483036 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:31:58.482943 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518108336" Jul 24 16:31:58.626695 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:31:58.626660 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-139787518108336"] Jul 24 16:31:58.628180 ip-10-0-141-58 kubenswrapper[2569]: W0724 16:31:58.628154 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3bf8d79_f9ec_45b9_a0a3_cd939c5dacb7.slice/crio-b765474e3cc0f2607eb2b28e12e8b7c46a3ad432f1c22cccd463fe344ffdf550 WatchSource:0}: Error finding container b765474e3cc0f2607eb2b28e12e8b7c46a3ad432f1c22cccd463fe344ffdf550: Status 404 returned error can't find the container with id b765474e3cc0f2607eb2b28e12e8b7c46a3ad432f1c22cccd463fe344ffdf550 Jul 24 16:31:58.629744 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:31:58.629719 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 16:31:58.942325 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:31:58.942292 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-139787518108336" event={"ID":"d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7","Type":"ContainerStarted","Data":"b765474e3cc0f2607eb2b28e12e8b7c46a3ad432f1c22cccd463fe344ffdf550"} Jul 24 16:31:59.950105 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:31:59.950065 2569 generic.go:356] "Generic (PLEG): container finished" podID="d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7" containerID="e0870e5512a48af5fd05e3c028a3f6bef79ae79d60dae05e260570270f5026e5" exitCode=0 Jul 24 16:31:59.950499 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:31:59.950157 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-139787518108336" event={"ID":"d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7","Type":"ContainerDied","Data":"e0870e5512a48af5fd05e3c028a3f6bef79ae79d60dae05e260570270f5026e5"} Jul 24 16:32:01.101948 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.101923 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518108336" Jul 24 16:32:01.162037 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.162001 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7462-139787518108336_d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7/test-curl-7462-139787518108336/0.log" Jul 24 16:32:01.188032 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.187996 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7462-139787518108336"] Jul 24 16:32:01.194293 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.194266 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7462-139787518108336"] Jul 24 16:32:01.224043 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.223971 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm749\" (UniqueName: \"kubernetes.io/projected/d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7-kube-api-access-zm749\") pod \"d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7\" (UID: \"d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7\") " Jul 24 16:32:01.226111 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.226088 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7-kube-api-access-zm749" (OuterVolumeSpecName: "kube-api-access-zm749") pod "d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7" (UID: "d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7"). InnerVolumeSpecName "kube-api-access-zm749". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:32:01.321332 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.321298 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7462-139787518108464"] Jul 24 16:32:01.321822 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.321807 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7" containerName="test-curl-7462-139787518108336" Jul 24 16:32:01.321876 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.321824 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7" containerName="test-curl-7462-139787518108336" Jul 24 16:32:01.321931 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.321922 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7" containerName="test-curl-7462-139787518108336" Jul 24 16:32:01.324721 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.324700 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm749\" (UniqueName: \"kubernetes.io/projected/d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7-kube-api-access-zm749\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:32:01.325175 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.325161 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518108464" Jul 24 16:32:01.330075 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.330046 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-139787518108464"] Jul 24 16:32:01.425679 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.425644 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r67q2\" (UniqueName: \"kubernetes.io/projected/b224a6e5-b45d-4613-acb1-add04aaf615a-kube-api-access-r67q2\") pod \"test-curl-7462-139787518108464\" (UID: \"b224a6e5-b45d-4613-acb1-add04aaf615a\") " pod="opendatahub/test-curl-7462-139787518108464" Jul 24 16:32:01.527024 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.526930 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r67q2\" (UniqueName: \"kubernetes.io/projected/b224a6e5-b45d-4613-acb1-add04aaf615a-kube-api-access-r67q2\") pod \"test-curl-7462-139787518108464\" (UID: \"b224a6e5-b45d-4613-acb1-add04aaf615a\") " pod="opendatahub/test-curl-7462-139787518108464" Jul 24 16:32:01.535805 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.535774 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r67q2\" (UniqueName: \"kubernetes.io/projected/b224a6e5-b45d-4613-acb1-add04aaf615a-kube-api-access-r67q2\") pod \"test-curl-7462-139787518108464\" (UID: \"b224a6e5-b45d-4613-acb1-add04aaf615a\") " pod="opendatahub/test-curl-7462-139787518108464" Jul 24 16:32:01.629680 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.629648 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7" path="/var/lib/kubelet/pods/d3bf8d79-f9ec-45b9-a0a3-cd939c5dacb7/volumes" Jul 24 16:32:01.636892 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.636843 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518108464" Jul 24 16:32:01.807532 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.807438 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-139787518108464"] Jul 24 16:32:01.810287 ip-10-0-141-58 kubenswrapper[2569]: W0724 16:32:01.810260 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb224a6e5_b45d_4613_acb1_add04aaf615a.slice/crio-24e97cf7954cf7bb1a4a54b14a721242d0df9949ebf8d39f64454ca7750b2a75 WatchSource:0}: Error finding container 24e97cf7954cf7bb1a4a54b14a721242d0df9949ebf8d39f64454ca7750b2a75: Status 404 returned error can't find the container with id 24e97cf7954cf7bb1a4a54b14a721242d0df9949ebf8d39f64454ca7750b2a75 Jul 24 16:32:01.963659 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.963631 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518108336" Jul 24 16:32:01.963866 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.963686 2569 scope.go:117] "RemoveContainer" containerID="e0870e5512a48af5fd05e3c028a3f6bef79ae79d60dae05e260570270f5026e5" Jul 24 16:32:01.965230 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:01.965185 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-139787518108464" event={"ID":"b224a6e5-b45d-4613-acb1-add04aaf615a","Type":"ContainerStarted","Data":"24e97cf7954cf7bb1a4a54b14a721242d0df9949ebf8d39f64454ca7750b2a75"} Jul 24 16:32:02.972509 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:02.972474 2569 generic.go:356] "Generic (PLEG): container finished" podID="b224a6e5-b45d-4613-acb1-add04aaf615a" containerID="c80ae177b54cd1ddf93847eb7bef25e559cff7c4008a97eb3b5685847582dcad" exitCode=0 Jul 24 16:32:02.972935 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:02.972559 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-139787518108464" event={"ID":"b224a6e5-b45d-4613-acb1-add04aaf615a","Type":"ContainerDied","Data":"c80ae177b54cd1ddf93847eb7bef25e559cff7c4008a97eb3b5685847582dcad"} Jul 24 16:32:04.130696 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.130671 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518108464" Jul 24 16:32:04.178452 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.178413 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7462-139787518108464_b224a6e5-b45d-4613-acb1-add04aaf615a/test-curl-7462-139787518108464/0.log" Jul 24 16:32:04.207327 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.207281 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7462-139787518108464"] Jul 24 16:32:04.213559 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.213516 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7462-139787518108464"] Jul 24 16:32:04.254110 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.254025 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r67q2\" (UniqueName: \"kubernetes.io/projected/b224a6e5-b45d-4613-acb1-add04aaf615a-kube-api-access-r67q2\") pod \"b224a6e5-b45d-4613-acb1-add04aaf615a\" (UID: \"b224a6e5-b45d-4613-acb1-add04aaf615a\") " Jul 24 16:32:04.256099 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.256074 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b224a6e5-b45d-4613-acb1-add04aaf615a-kube-api-access-r67q2" (OuterVolumeSpecName: "kube-api-access-r67q2") pod "b224a6e5-b45d-4613-acb1-add04aaf615a" (UID: "b224a6e5-b45d-4613-acb1-add04aaf615a"). InnerVolumeSpecName "kube-api-access-r67q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:32:04.354770 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.354726 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r67q2\" (UniqueName: \"kubernetes.io/projected/b224a6e5-b45d-4613-acb1-add04aaf615a-kube-api-access-r67q2\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:32:04.399610 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.399576 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7462-139787518071840"] Jul 24 16:32:04.400160 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.400140 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b224a6e5-b45d-4613-acb1-add04aaf615a" containerName="test-curl-7462-139787518108464" Jul 24 16:32:04.400306 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.400163 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b224a6e5-b45d-4613-acb1-add04aaf615a" containerName="test-curl-7462-139787518108464" Jul 24 16:32:04.400377 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.400310 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b224a6e5-b45d-4613-acb1-add04aaf615a" containerName="test-curl-7462-139787518108464" Jul 24 16:32:04.403367 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.403347 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518071840" Jul 24 16:32:04.420739 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.420711 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-139787518071840"] Jul 24 16:32:04.556921 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.556822 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzvzn\" (UniqueName: \"kubernetes.io/projected/8b02c335-b2d4-4a2c-84a4-6b30e5041102-kube-api-access-tzvzn\") pod \"test-curl-7462-139787518071840\" (UID: \"8b02c335-b2d4-4a2c-84a4-6b30e5041102\") " pod="opendatahub/test-curl-7462-139787518071840" Jul 24 16:32:04.657560 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.657520 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzvzn\" (UniqueName: \"kubernetes.io/projected/8b02c335-b2d4-4a2c-84a4-6b30e5041102-kube-api-access-tzvzn\") pod \"test-curl-7462-139787518071840\" (UID: \"8b02c335-b2d4-4a2c-84a4-6b30e5041102\") " pod="opendatahub/test-curl-7462-139787518071840" Jul 24 16:32:04.666250 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.666186 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzvzn\" (UniqueName: \"kubernetes.io/projected/8b02c335-b2d4-4a2c-84a4-6b30e5041102-kube-api-access-tzvzn\") pod \"test-curl-7462-139787518071840\" (UID: \"8b02c335-b2d4-4a2c-84a4-6b30e5041102\") " pod="opendatahub/test-curl-7462-139787518071840" Jul 24 16:32:04.713281 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.713245 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518071840" Jul 24 16:32:04.886576 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.886550 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-139787518071840"] Jul 24 16:32:04.887891 ip-10-0-141-58 kubenswrapper[2569]: W0724 16:32:04.887864 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b02c335_b2d4_4a2c_84a4_6b30e5041102.slice/crio-d86b4731dd70e33c731890d455ff7892b9ef0cd66abe1aab770caa25f2bf675e WatchSource:0}: Error finding container d86b4731dd70e33c731890d455ff7892b9ef0cd66abe1aab770caa25f2bf675e: Status 404 returned error can't find the container with id d86b4731dd70e33c731890d455ff7892b9ef0cd66abe1aab770caa25f2bf675e Jul 24 16:32:04.986186 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.986148 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-139787518071840" event={"ID":"8b02c335-b2d4-4a2c-84a4-6b30e5041102","Type":"ContainerStarted","Data":"d86b4731dd70e33c731890d455ff7892b9ef0cd66abe1aab770caa25f2bf675e"} Jul 24 16:32:04.987515 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.987487 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24e97cf7954cf7bb1a4a54b14a721242d0df9949ebf8d39f64454ca7750b2a75" Jul 24 16:32:04.987635 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:04.987527 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518108464" Jul 24 16:32:04.998623 ip-10-0-141-58 kubenswrapper[2569]: E0724 16:32:04.998591 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7462-139787518108464\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="b224a6e5-b45d-4613-acb1-add04aaf615a" pod="opendatahub/test-curl-7462-139787518108464" Jul 24 16:32:05.628566 ip-10-0-141-58 kubenswrapper[2569]: E0724 16:32:05.628471 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7462-139787518108464\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="b224a6e5-b45d-4613-acb1-add04aaf615a" pod="opendatahub/test-curl-7462-139787518108464" Jul 24 16:32:05.630134 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:05.630108 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b224a6e5-b45d-4613-acb1-add04aaf615a" path="/var/lib/kubelet/pods/b224a6e5-b45d-4613-acb1-add04aaf615a/volumes" Jul 24 16:32:05.996184 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:05.996153 2569 generic.go:356] "Generic (PLEG): container finished" podID="8b02c335-b2d4-4a2c-84a4-6b30e5041102" containerID="bb90fb4a928581e44a78385d0d22ea319a8d5fd0d027e47fb009eb9378110aab" exitCode=0 Jul 24 16:32:05.996390 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:05.996200 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-139787518071840" event={"ID":"8b02c335-b2d4-4a2c-84a4-6b30e5041102","Type":"ContainerDied","Data":"bb90fb4a928581e44a78385d0d22ea319a8d5fd0d027e47fb009eb9378110aab"} Jul 24 16:32:07.152188 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:07.152164 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518071840" Jul 24 16:32:07.174874 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:07.174846 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7462-139787518071840_8b02c335-b2d4-4a2c-84a4-6b30e5041102/test-curl-7462-139787518071840/0.log" Jul 24 16:32:07.198906 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:07.198873 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7462-139787518071840"] Jul 24 16:32:07.201479 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:07.201459 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7462-139787518071840"] Jul 24 16:32:07.284294 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:07.284161 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzvzn\" (UniqueName: \"kubernetes.io/projected/8b02c335-b2d4-4a2c-84a4-6b30e5041102-kube-api-access-tzvzn\") pod \"8b02c335-b2d4-4a2c-84a4-6b30e5041102\" (UID: \"8b02c335-b2d4-4a2c-84a4-6b30e5041102\") " Jul 24 16:32:07.286348 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:07.286320 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b02c335-b2d4-4a2c-84a4-6b30e5041102-kube-api-access-tzvzn" (OuterVolumeSpecName: "kube-api-access-tzvzn") pod "8b02c335-b2d4-4a2c-84a4-6b30e5041102" (UID: "8b02c335-b2d4-4a2c-84a4-6b30e5041102"). InnerVolumeSpecName "kube-api-access-tzvzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:32:07.327247 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:07.327196 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7462-139787518071840"] Jul 24 16:32:07.327701 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:07.327686 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b02c335-b2d4-4a2c-84a4-6b30e5041102" containerName="test-curl-7462-139787518071840" Jul 24 16:32:07.327750 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:07.327703 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b02c335-b2d4-4a2c-84a4-6b30e5041102" containerName="test-curl-7462-139787518071840" Jul 24 16:32:07.327792 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:07.327783 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b02c335-b2d4-4a2c-84a4-6b30e5041102" containerName="test-curl-7462-139787518071840" Jul 24 16:32:07.330964 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:07.330947 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518071840" Jul 24 16:32:07.336813 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:07.336786 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-139787518071840"] Jul 24 16:32:07.385638 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:07.385603 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzvzn\" (UniqueName: \"kubernetes.io/projected/8b02c335-b2d4-4a2c-84a4-6b30e5041102-kube-api-access-tzvzn\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:32:07.486752 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:07.486720 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5x2q\" (UniqueName: \"kubernetes.io/projected/72411784-f444-4bb1-8ff8-f551c5a5d651-kube-api-access-z5x2q\") pod \"test-curl-7462-139787518071840\" (UID: \"72411784-f444-4bb1-8ff8-f551c5a5d651\") " pod="opendatahub/test-curl-7462-139787518071840" Jul 24 16:32:07.587621 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:07.587533 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5x2q\" (UniqueName: \"kubernetes.io/projected/72411784-f444-4bb1-8ff8-f551c5a5d651-kube-api-access-z5x2q\") pod \"test-curl-7462-139787518071840\" (UID: \"72411784-f444-4bb1-8ff8-f551c5a5d651\") " pod="opendatahub/test-curl-7462-139787518071840" Jul 24 16:32:07.596744 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:07.596717 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5x2q\" (UniqueName: \"kubernetes.io/projected/72411784-f444-4bb1-8ff8-f551c5a5d651-kube-api-access-z5x2q\") pod \"test-curl-7462-139787518071840\" (UID: \"72411784-f444-4bb1-8ff8-f551c5a5d651\") " pod="opendatahub/test-curl-7462-139787518071840" Jul 24 16:32:07.630424 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:07.630390 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b02c335-b2d4-4a2c-84a4-6b30e5041102" path="/var/lib/kubelet/pods/8b02c335-b2d4-4a2c-84a4-6b30e5041102/volumes" Jul 24 16:32:07.643944 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:07.643899 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518071840" Jul 24 16:32:07.791024 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:07.790991 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-139787518071840"] Jul 24 16:32:07.792534 ip-10-0-141-58 kubenswrapper[2569]: W0724 16:32:07.792507 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72411784_f444_4bb1_8ff8_f551c5a5d651.slice/crio-34cc3415bc426871eb6d86df91428c328058e6ae2c1ee88793fdf601cd06e45a WatchSource:0}: Error finding container 34cc3415bc426871eb6d86df91428c328058e6ae2c1ee88793fdf601cd06e45a: Status 404 returned error can't find the container with id 34cc3415bc426871eb6d86df91428c328058e6ae2c1ee88793fdf601cd06e45a Jul 24 16:32:08.009659 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:08.009617 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-139787518071840" event={"ID":"72411784-f444-4bb1-8ff8-f551c5a5d651","Type":"ContainerStarted","Data":"34cc3415bc426871eb6d86df91428c328058e6ae2c1ee88793fdf601cd06e45a"} Jul 24 16:32:08.010868 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:08.010846 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518071840" Jul 24 16:32:08.011021 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:08.010876 2569 scope.go:117] "RemoveContainer" containerID="bb90fb4a928581e44a78385d0d22ea319a8d5fd0d027e47fb009eb9378110aab" Jul 24 16:32:09.018568 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:09.018535 2569 generic.go:356] "Generic (PLEG): container finished" podID="72411784-f444-4bb1-8ff8-f551c5a5d651" containerID="e1ff5dbefdcdfb315ea4bcf4abcdddfd697ac817d3e64eb6d58fc24b29da9702" exitCode=0 Jul 24 16:32:09.019015 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:09.018623 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-139787518071840" event={"ID":"72411784-f444-4bb1-8ff8-f551c5a5d651","Type":"ContainerDied","Data":"e1ff5dbefdcdfb315ea4bcf4abcdddfd697ac817d3e64eb6d58fc24b29da9702"} Jul 24 16:32:10.173877 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:10.173846 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518071840" Jul 24 16:32:10.196698 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:10.196620 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7462-139787518071840_72411784-f444-4bb1-8ff8-f551c5a5d651/test-curl-7462-139787518071840/0.log" Jul 24 16:32:10.224382 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:10.224345 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7462-139787518071840"] Jul 24 16:32:10.227662 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:10.227628 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7462-139787518071840"] Jul 24 16:32:10.315389 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:10.315293 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5x2q\" (UniqueName: \"kubernetes.io/projected/72411784-f444-4bb1-8ff8-f551c5a5d651-kube-api-access-z5x2q\") pod \"72411784-f444-4bb1-8ff8-f551c5a5d651\" (UID: \"72411784-f444-4bb1-8ff8-f551c5a5d651\") " Jul 24 16:32:10.317448 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:10.317423 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72411784-f444-4bb1-8ff8-f551c5a5d651-kube-api-access-z5x2q" (OuterVolumeSpecName: "kube-api-access-z5x2q") pod "72411784-f444-4bb1-8ff8-f551c5a5d651" (UID: "72411784-f444-4bb1-8ff8-f551c5a5d651"). InnerVolumeSpecName "kube-api-access-z5x2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:32:10.355361 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:10.355325 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7462-139787518073280"] Jul 24 16:32:10.355873 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:10.355855 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72411784-f444-4bb1-8ff8-f551c5a5d651" containerName="test-curl-7462-139787518071840" Jul 24 16:32:10.355987 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:10.355875 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="72411784-f444-4bb1-8ff8-f551c5a5d651" containerName="test-curl-7462-139787518071840" Jul 24 16:32:10.356076 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:10.356060 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="72411784-f444-4bb1-8ff8-f551c5a5d651" containerName="test-curl-7462-139787518071840" Jul 24 16:32:10.359272 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:10.359251 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518073280" Jul 24 16:32:10.364410 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:10.364377 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-139787518073280"] Jul 24 16:32:10.416616 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:10.416579 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5x2q\" (UniqueName: \"kubernetes.io/projected/72411784-f444-4bb1-8ff8-f551c5a5d651-kube-api-access-z5x2q\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:32:10.517733 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:10.517698 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqn2m\" (UniqueName: \"kubernetes.io/projected/3adad0e0-9e04-4635-8c55-ee95a339f710-kube-api-access-qqn2m\") pod \"test-curl-7462-139787518073280\" (UID: \"3adad0e0-9e04-4635-8c55-ee95a339f710\") " pod="opendatahub/test-curl-7462-139787518073280" Jul 24 16:32:10.619315 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:10.619197 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqn2m\" (UniqueName: \"kubernetes.io/projected/3adad0e0-9e04-4635-8c55-ee95a339f710-kube-api-access-qqn2m\") pod \"test-curl-7462-139787518073280\" (UID: \"3adad0e0-9e04-4635-8c55-ee95a339f710\") " pod="opendatahub/test-curl-7462-139787518073280" Jul 24 16:32:10.628050 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:10.628010 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqn2m\" (UniqueName: \"kubernetes.io/projected/3adad0e0-9e04-4635-8c55-ee95a339f710-kube-api-access-qqn2m\") pod \"test-curl-7462-139787518073280\" (UID: \"3adad0e0-9e04-4635-8c55-ee95a339f710\") " pod="opendatahub/test-curl-7462-139787518073280" Jul 24 16:32:10.670933 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:10.670893 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518073280" Jul 24 16:32:10.813825 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:10.813798 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7462-139787518073280"] Jul 24 16:32:10.815104 ip-10-0-141-58 kubenswrapper[2569]: W0724 16:32:10.815077 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3adad0e0_9e04_4635_8c55_ee95a339f710.slice/crio-5da714961b25e69a0ff581a5c8c9b9a44514d4d22bb8cd63e5828f64b51c8090 WatchSource:0}: Error finding container 5da714961b25e69a0ff581a5c8c9b9a44514d4d22bb8cd63e5828f64b51c8090: Status 404 returned error can't find the container with id 5da714961b25e69a0ff581a5c8c9b9a44514d4d22bb8cd63e5828f64b51c8090 Jul 24 16:32:11.035957 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:11.035912 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-139787518073280" event={"ID":"3adad0e0-9e04-4635-8c55-ee95a339f710","Type":"ContainerStarted","Data":"5da714961b25e69a0ff581a5c8c9b9a44514d4d22bb8cd63e5828f64b51c8090"} Jul 24 16:32:11.037273 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:11.037250 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518071840" Jul 24 16:32:11.037273 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:11.037268 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34cc3415bc426871eb6d86df91428c328058e6ae2c1ee88793fdf601cd06e45a" Jul 24 16:32:11.048280 ip-10-0-141-58 kubenswrapper[2569]: E0724 16:32:11.048241 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7462-139787518071840\" is forbidden: User \"system:node:ip-10-0-141-58.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-58.ec2.internal' and this object" podUID="72411784-f444-4bb1-8ff8-f551c5a5d651" pod="opendatahub/test-curl-7462-139787518071840" Jul 24 16:32:11.629598 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:11.629564 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72411784-f444-4bb1-8ff8-f551c5a5d651" path="/var/lib/kubelet/pods/72411784-f444-4bb1-8ff8-f551c5a5d651/volumes" Jul 24 16:32:12.046203 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:12.046168 2569 generic.go:356] "Generic (PLEG): container finished" podID="3adad0e0-9e04-4635-8c55-ee95a339f710" containerID="082bf8fab75e9f00b226ce85ad27a809f5e572b8803c98ab991b532d3983aab0" exitCode=0 Jul 24 16:32:12.046395 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:12.046263 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7462-139787518073280" event={"ID":"3adad0e0-9e04-4635-8c55-ee95a339f710","Type":"ContainerDied","Data":"082bf8fab75e9f00b226ce85ad27a809f5e572b8803c98ab991b532d3983aab0"} Jul 24 16:32:13.217700 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:13.217673 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518073280" Jul 24 16:32:13.239091 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:13.239053 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7462-139787518073280_3adad0e0-9e04-4635-8c55-ee95a339f710/test-curl-7462-139787518073280/0.log" Jul 24 16:32:13.263890 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:13.263858 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7462-139787518073280"] Jul 24 16:32:13.268443 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:13.268413 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7462-139787518073280"] Jul 24 16:32:13.353428 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:13.353334 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqn2m\" (UniqueName: \"kubernetes.io/projected/3adad0e0-9e04-4635-8c55-ee95a339f710-kube-api-access-qqn2m\") pod \"3adad0e0-9e04-4635-8c55-ee95a339f710\" (UID: \"3adad0e0-9e04-4635-8c55-ee95a339f710\") " Jul 24 16:32:13.355592 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:13.355557 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3adad0e0-9e04-4635-8c55-ee95a339f710-kube-api-access-qqn2m" (OuterVolumeSpecName: "kube-api-access-qqn2m") pod "3adad0e0-9e04-4635-8c55-ee95a339f710" (UID: "3adad0e0-9e04-4635-8c55-ee95a339f710"). InnerVolumeSpecName "kube-api-access-qqn2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:32:13.454249 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:13.454205 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqn2m\" (UniqueName: \"kubernetes.io/projected/3adad0e0-9e04-4635-8c55-ee95a339f710-kube-api-access-qqn2m\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:32:13.629889 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:13.629808 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3adad0e0-9e04-4635-8c55-ee95a339f710" path="/var/lib/kubelet/pods/3adad0e0-9e04-4635-8c55-ee95a339f710/volumes" Jul 24 16:32:14.063165 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:14.063139 2569 scope.go:117] "RemoveContainer" containerID="082bf8fab75e9f00b226ce85ad27a809f5e572b8803c98ab991b532d3983aab0" Jul 24 16:32:14.063165 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:14.063153 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7462-139787518073280" Jul 24 16:32:20.753825 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:20.753791 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg"] Jul 24 16:32:20.754457 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:20.754426 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3adad0e0-9e04-4635-8c55-ee95a339f710" containerName="test-curl-7462-139787518073280" Jul 24 16:32:20.754457 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:20.754452 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3adad0e0-9e04-4635-8c55-ee95a339f710" containerName="test-curl-7462-139787518073280" Jul 24 16:32:20.754698 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:20.754678 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3adad0e0-9e04-4635-8c55-ee95a339f710" containerName="test-curl-7462-139787518073280" Jul 24 16:32:20.759615 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:20.759595 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" Jul 24 16:32:20.764413 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:20.764388 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-0107ee-serving-cert\"" Jul 24 16:32:20.770779 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:20.770758 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg"] Jul 24 16:32:20.822449 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:20.822417 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6zt2\" (UniqueName: \"kubernetes.io/projected/9924f396-4ec9-41df-8ade-6957abfb3287-kube-api-access-w6zt2\") pod \"maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg\" (UID: \"9924f396-4ec9-41df-8ade-6957abfb3287\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" Jul 24 16:32:20.822610 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:20.822473 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9924f396-4ec9-41df-8ade-6957abfb3287-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg\" (UID: \"9924f396-4ec9-41df-8ade-6957abfb3287\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" Jul 24 16:32:20.822610 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:20.822529 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9924f396-4ec9-41df-8ade-6957abfb3287-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg\" (UID: \"9924f396-4ec9-41df-8ade-6957abfb3287\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" Jul 24 16:32:20.923243 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:20.923155 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9924f396-4ec9-41df-8ade-6957abfb3287-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg\" (UID: \"9924f396-4ec9-41df-8ade-6957abfb3287\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" Jul 24 16:32:20.923423 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:20.923307 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6zt2\" (UniqueName: \"kubernetes.io/projected/9924f396-4ec9-41df-8ade-6957abfb3287-kube-api-access-w6zt2\") pod \"maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg\" (UID: \"9924f396-4ec9-41df-8ade-6957abfb3287\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" Jul 24 16:32:20.923423 ip-10-0-141-58 kubenswrapper[2569]: E0724 16:32:20.923330 2569 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-serving-cert: secret "maas-api-e2e-ipp-a-0107ee-serving-cert" not found Jul 24 16:32:20.923423 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:20.923374 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9924f396-4ec9-41df-8ade-6957abfb3287-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg\" (UID: \"9924f396-4ec9-41df-8ade-6957abfb3287\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" Jul 24 16:32:20.923423 ip-10-0-141-58 kubenswrapper[2569]: E0724 16:32:20.923402 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9924f396-4ec9-41df-8ade-6957abfb3287-maas-api-tls podName:9924f396-4ec9-41df-8ade-6957abfb3287 nodeName:}" failed. No retries permitted until 2026-07-24 16:32:21.423386642 +0000 UTC m=+2816.341356924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9924f396-4ec9-41df-8ade-6957abfb3287-maas-api-tls") pod "maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" (UID: "9924f396-4ec9-41df-8ade-6957abfb3287") : secret "maas-api-e2e-ipp-a-0107ee-serving-cert" not found Jul 24 16:32:20.924014 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:20.923993 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9924f396-4ec9-41df-8ade-6957abfb3287-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg\" (UID: \"9924f396-4ec9-41df-8ade-6957abfb3287\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" Jul 24 16:32:20.932283 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:20.932249 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6zt2\" (UniqueName: \"kubernetes.io/projected/9924f396-4ec9-41df-8ade-6957abfb3287-kube-api-access-w6zt2\") pod \"maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg\" (UID: \"9924f396-4ec9-41df-8ade-6957abfb3287\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" Jul 24 16:32:21.427630 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:21.427587 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9924f396-4ec9-41df-8ade-6957abfb3287-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg\" (UID: \"9924f396-4ec9-41df-8ade-6957abfb3287\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" Jul 24 16:32:21.430719 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:21.430688 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9924f396-4ec9-41df-8ade-6957abfb3287-maas-api-tls\") pod \"maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg\" (UID: \"9924f396-4ec9-41df-8ade-6957abfb3287\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" Jul 24 16:32:21.672454 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:21.672420 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" Jul 24 16:32:21.821407 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:21.821375 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg"] Jul 24 16:32:21.822645 ip-10-0-141-58 kubenswrapper[2569]: W0724 16:32:21.822620 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9924f396_4ec9_41df_8ade_6957abfb3287.slice/crio-db4af39e58aff57f3ba6390f62c6fa984f383e714b180a5ccc2fee20230de143 WatchSource:0}: Error finding container db4af39e58aff57f3ba6390f62c6fa984f383e714b180a5ccc2fee20230de143: Status 404 returned error can't find the container with id db4af39e58aff57f3ba6390f62c6fa984f383e714b180a5ccc2fee20230de143 Jul 24 16:32:22.118391 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:22.118296 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" event={"ID":"9924f396-4ec9-41df-8ade-6957abfb3287","Type":"ContainerStarted","Data":"db4af39e58aff57f3ba6390f62c6fa984f383e714b180a5ccc2fee20230de143"} Jul 24 16:32:23.127681 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:23.127640 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" event={"ID":"9924f396-4ec9-41df-8ade-6957abfb3287","Type":"ContainerStarted","Data":"c0f2ccbadb6df73a2fc81be51865c8f3c93aa42037327de31cd90741e1528fec"} Jul 24 16:32:23.128113 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:23.127792 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" Jul 24 16:32:23.146374 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:23.146315 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" podStartSLOduration=2.6655219089999997 podStartE2EDuration="3.146299854s" totalImagesPullingTime="480.777945ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:32:23.144630896 +0000 UTC m=+2818.062601197" watchObservedRunningTime="2026-07-24 16:32:23.146299854 +0000 UTC m=+2818.064270153" Jul 24 16:32:29.139784 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:29.139752 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" Jul 24 16:32:40.589591 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:40.589553 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd"] Jul 24 16:32:40.593550 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:40.593531 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd" Jul 24 16:32:40.596692 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:40.596667 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-52d140-dockercfg-2lnmc\"" Jul 24 16:32:40.596979 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:40.596963 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-52d140\"" Jul 24 16:32:40.607811 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:40.607781 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd"] Jul 24 16:32:40.713112 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:40.713074 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150a0264-5b33-49c3-ba4b-4e6bae786f86-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd\" (UID: \"150a0264-5b33-49c3-ba4b-4e6bae786f86\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd" Jul 24 16:32:40.713112 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:40.713119 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsnhf\" (UniqueName: \"kubernetes.io/projected/150a0264-5b33-49c3-ba4b-4e6bae786f86-kube-api-access-gsnhf\") pod \"payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd\" (UID: \"150a0264-5b33-49c3-ba4b-4e6bae786f86\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd" Jul 24 16:32:40.814693 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:40.814653 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150a0264-5b33-49c3-ba4b-4e6bae786f86-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd\" (UID: \"150a0264-5b33-49c3-ba4b-4e6bae786f86\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd" Jul 24 16:32:40.814693 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:40.814699 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsnhf\" (UniqueName: \"kubernetes.io/projected/150a0264-5b33-49c3-ba4b-4e6bae786f86-kube-api-access-gsnhf\") pod \"payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd\" (UID: \"150a0264-5b33-49c3-ba4b-4e6bae786f86\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd" Jul 24 16:32:40.815401 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:40.815384 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150a0264-5b33-49c3-ba4b-4e6bae786f86-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd\" (UID: \"150a0264-5b33-49c3-ba4b-4e6bae786f86\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd" Jul 24 16:32:40.823924 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:40.823898 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsnhf\" (UniqueName: \"kubernetes.io/projected/150a0264-5b33-49c3-ba4b-4e6bae786f86-kube-api-access-gsnhf\") pod \"payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd\" (UID: \"150a0264-5b33-49c3-ba4b-4e6bae786f86\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd" Jul 24 16:32:40.904976 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:40.904945 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd" Jul 24 16:32:41.052433 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:41.052407 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd"] Jul 24 16:32:41.053690 ip-10-0-141-58 kubenswrapper[2569]: W0724 16:32:41.053659 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150a0264_5b33_49c3_ba4b_4e6bae786f86.slice/crio-fe4f309f9eb7eed1e2ccfb00d579be988e53d07b04b063fcd54cad36d2ac716d WatchSource:0}: Error finding container fe4f309f9eb7eed1e2ccfb00d579be988e53d07b04b063fcd54cad36d2ac716d: Status 404 returned error can't find the container with id fe4f309f9eb7eed1e2ccfb00d579be988e53d07b04b063fcd54cad36d2ac716d Jul 24 16:32:41.244418 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:41.244333 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd" event={"ID":"150a0264-5b33-49c3-ba4b-4e6bae786f86","Type":"ContainerStarted","Data":"c0400a002fffb6451b2b9c63bf405a38a9c7224fc8bcb1437634a62283baa65c"} Jul 24 16:32:41.244418 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:41.244390 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd" event={"ID":"150a0264-5b33-49c3-ba4b-4e6bae786f86","Type":"ContainerStarted","Data":"fe4f309f9eb7eed1e2ccfb00d579be988e53d07b04b063fcd54cad36d2ac716d"} Jul 24 16:32:41.244664 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:41.244470 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd" Jul 24 16:32:41.277109 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:41.277046 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd" podStartSLOduration=1.277030191 podStartE2EDuration="1.277030191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:32:41.273928372 +0000 UTC m=+2836.191898685" watchObservedRunningTime="2026-07-24 16:32:41.277030191 +0000 UTC m=+2836.195000537" Jul 24 16:32:52.253732 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:32:52.253702 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd" Jul 24 16:33:01.640349 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:01.640315 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-kbzx2_ca5f962c-6f6a-4360-874c-cadf65694353/payload-processing/0.log" Jul 24 16:33:01.775719 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:01.775693 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd_150a0264-5b33-49c3-ba4b-4e6bae786f86/payload-processing/0.log" Jul 24 16:33:43.893431 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:43.893350 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-kbzx2_ca5f962c-6f6a-4360-874c-cadf65694353/payload-processing/0.log" Jul 24 16:33:44.023758 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.023731 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd_150a0264-5b33-49c3-ba4b-4e6bae786f86/payload-processing/0.log" Jul 24 16:33:44.645267 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.645239 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg"] Jul 24 16:33:44.649101 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.649085 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.651604 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.651583 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 16:33:44.651711 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.651693 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-2ba900-openshift-default-dockercfg-gpqxz\"" Jul 24 16:33:44.666874 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.666840 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg"] Jul 24 16:33:44.751340 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.751304 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-workload-certs\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.751513 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.751374 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-workload-socket\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.751513 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.751424 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dxnl\" (UniqueName: \"kubernetes.io/projected/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-kube-api-access-9dxnl\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.751513 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.751447 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-credential-socket\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.751513 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.751488 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-data\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.751654 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.751521 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-envoy\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.751654 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.751547 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.751654 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.751595 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-podinfo\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.751654 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.751619 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-token\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.852206 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.852168 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-workload-socket\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.852428 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.852259 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dxnl\" (UniqueName: \"kubernetes.io/projected/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-kube-api-access-9dxnl\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.852428 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.852299 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-credential-socket\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.852428 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.852320 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-data\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.852428 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.852351 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-envoy\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.852428 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.852380 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.852697 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.852436 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-podinfo\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.852697 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.852467 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-token\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.852697 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.852564 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-workload-certs\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.852697 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.852635 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-workload-socket\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.852697 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.852660 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-credential-socket\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.852974 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.852949 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-data\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.852974 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.852957 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-workload-certs\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.853155 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.853135 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istiod-ca-cert\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.854739 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.854712 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-envoy\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.854875 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.854856 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-podinfo\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.865508 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.865485 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-token\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.865706 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.865686 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dxnl\" (UniqueName: \"kubernetes.io/projected/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-kube-api-access-9dxnl\") pod \"e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:44.960827 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:44.960742 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:45.112902 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:45.112871 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg"] Jul 24 16:33:45.113689 ip-10-0-141-58 kubenswrapper[2569]: W0724 16:33:45.113665 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca37a3b3_b985_4e77_a9b2_009e94ca4d3c.slice/crio-656c207dcf152b51cd1edf23c6a451e748e4c83ec33501f05efb4a92f60c5928 WatchSource:0}: Error finding container 656c207dcf152b51cd1edf23c6a451e748e4c83ec33501f05efb4a92f60c5928: Status 404 returned error can't find the container with id 656c207dcf152b51cd1edf23c6a451e748e4c83ec33501f05efb4a92f60c5928 Jul 24 16:33:45.652143 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:45.652113 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" event={"ID":"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c","Type":"ContainerStarted","Data":"656c207dcf152b51cd1edf23c6a451e748e4c83ec33501f05efb4a92f60c5928"} Jul 24 16:33:47.926991 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:47.926951 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 16:33:47.927306 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:47.927028 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 16:33:47.927306 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:47.927062 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 16:33:48.678786 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:48.678752 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" event={"ID":"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c","Type":"ContainerStarted","Data":"06c07dc8621a4a35ce5ed18416ba9516286075c1ba2df0900e2fdb66274678d2"} Jul 24 16:33:48.702791 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:48.702742 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" podStartSLOduration=1.891945559 podStartE2EDuration="4.702728217s" totalImagesPullingTime="2.810782658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:33:48.698776529 +0000 UTC m=+2903.616746829" watchObservedRunningTime="2026-07-24 16:33:48.702728217 +0000 UTC m=+2903.620698518" Jul 24 16:33:48.962005 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:48.961906 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:48.967298 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:48.967273 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:49.685609 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:49.685581 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:49.686527 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:49.686508 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:33:51.386041 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:51.386004 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd"] Jul 24 16:33:51.390707 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:51.390684 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd" Jul 24 16:33:51.393435 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:51.393415 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-2ba900-dockercfg-658bb\"" Jul 24 16:33:51.393616 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:51.393454 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-2ba900\"" Jul 24 16:33:51.399272 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:51.398667 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd"] Jul 24 16:33:51.412616 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:51.412587 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktdr7\" (UniqueName: \"kubernetes.io/projected/e41af8fc-47f1-428a-ad7b-6e5e17e69fd4-kube-api-access-ktdr7\") pod \"payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd\" (UID: \"e41af8fc-47f1-428a-ad7b-6e5e17e69fd4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd" Jul 24 16:33:51.412755 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:51.412703 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41af8fc-47f1-428a-ad7b-6e5e17e69fd4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd\" (UID: \"e41af8fc-47f1-428a-ad7b-6e5e17e69fd4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd" Jul 24 16:33:51.514251 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:51.514187 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktdr7\" (UniqueName: \"kubernetes.io/projected/e41af8fc-47f1-428a-ad7b-6e5e17e69fd4-kube-api-access-ktdr7\") pod \"payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd\" (UID: \"e41af8fc-47f1-428a-ad7b-6e5e17e69fd4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd" Jul 24 16:33:51.514426 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:51.514286 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41af8fc-47f1-428a-ad7b-6e5e17e69fd4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd\" (UID: \"e41af8fc-47f1-428a-ad7b-6e5e17e69fd4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd" Jul 24 16:33:51.515071 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:51.515050 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41af8fc-47f1-428a-ad7b-6e5e17e69fd4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd\" (UID: \"e41af8fc-47f1-428a-ad7b-6e5e17e69fd4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd" Jul 24 16:33:51.523116 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:51.523088 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktdr7\" (UniqueName: \"kubernetes.io/projected/e41af8fc-47f1-428a-ad7b-6e5e17e69fd4-kube-api-access-ktdr7\") pod \"payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd\" (UID: \"e41af8fc-47f1-428a-ad7b-6e5e17e69fd4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd" Jul 24 16:33:51.704384 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:51.704352 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd" Jul 24 16:33:51.859852 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:51.859817 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd"] Jul 24 16:33:51.860529 ip-10-0-141-58 kubenswrapper[2569]: W0724 16:33:51.860500 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41af8fc_47f1_428a_ad7b_6e5e17e69fd4.slice/crio-66a53a81e8a59dd8c05b9679b9769a25b572f3da8beb8268538c565e992a5d2c WatchSource:0}: Error finding container 66a53a81e8a59dd8c05b9679b9769a25b572f3da8beb8268538c565e992a5d2c: Status 404 returned error can't find the container with id 66a53a81e8a59dd8c05b9679b9769a25b572f3da8beb8268538c565e992a5d2c Jul 24 16:33:52.706529 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:52.706494 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd" event={"ID":"e41af8fc-47f1-428a-ad7b-6e5e17e69fd4","Type":"ContainerStarted","Data":"f23150b4eb1464a7593428aa3155d344e939af4f409c602b840b3e6900d81216"} Jul 24 16:33:52.706924 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:52.706547 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd" event={"ID":"e41af8fc-47f1-428a-ad7b-6e5e17e69fd4","Type":"ContainerStarted","Data":"66a53a81e8a59dd8c05b9679b9769a25b572f3da8beb8268538c565e992a5d2c"} Jul 24 16:33:52.706924 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:52.706573 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd" Jul 24 16:33:52.724980 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:33:52.724931 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd" podStartSLOduration=1.724917097 podStartE2EDuration="1.724917097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:33:52.723578714 +0000 UTC m=+2907.641549013" watchObservedRunningTime="2026-07-24 16:33:52.724917097 +0000 UTC m=+2907.642887397" Jul 24 16:34:03.716130 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:03.716094 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd" Jul 24 16:34:13.854224 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:13.854174 2569 generic.go:356] "Generic (PLEG): container finished" podID="ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" containerID="06c07dc8621a4a35ce5ed18416ba9516286075c1ba2df0900e2fdb66274678d2" exitCode=0 Jul 24 16:34:13.854736 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:13.854245 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" event={"ID":"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c","Type":"ContainerDied","Data":"06c07dc8621a4a35ce5ed18416ba9516286075c1ba2df0900e2fdb66274678d2"} Jul 24 16:34:13.854736 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:13.854657 2569 scope.go:117] "RemoveContainer" containerID="06c07dc8621a4a35ce5ed18416ba9516286075c1ba2df0900e2fdb66274678d2" Jul 24 16:34:13.855834 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:13.855796 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 16:34:13.855970 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:13.855894 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 16:34:13.856036 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:13.855965 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 16:34:14.862896 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:14.862859 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" event={"ID":"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c","Type":"ContainerStarted","Data":"83481c8064a69fc8a634f8f2fa3ae5488f3d8f7a1a6a6a03ebc8ce91e1cf31a0"} Jul 24 16:34:14.961630 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:14.961602 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:34:14.962622 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:14.962597 2569 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Jul 24 16:34:14.962754 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:14.962642 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" podUID="ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Jul 24 16:34:15.961980 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:15.961939 2569 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Jul 24 16:34:15.962444 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:15.962034 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" podUID="ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Jul 24 16:34:16.961866 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:16.961401 2569 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Jul 24 16:34:16.961866 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:16.961464 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" podUID="ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Jul 24 16:34:16.964935 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:16.964907 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd"] Jul 24 16:34:16.965193 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:16.965146 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd" podUID="e41af8fc-47f1-428a-ad7b-6e5e17e69fd4" containerName="payload-processing" containerID="cri-o://f23150b4eb1464a7593428aa3155d344e939af4f409c602b840b3e6900d81216" gracePeriod=30 Jul 24 16:34:17.284939 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:17.284903 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd" Jul 24 16:34:17.380820 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:17.380788 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41af8fc-47f1-428a-ad7b-6e5e17e69fd4-plugins-config-volume\") pod \"e41af8fc-47f1-428a-ad7b-6e5e17e69fd4\" (UID: \"e41af8fc-47f1-428a-ad7b-6e5e17e69fd4\") " Jul 24 16:34:17.381006 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:17.380843 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktdr7\" (UniqueName: \"kubernetes.io/projected/e41af8fc-47f1-428a-ad7b-6e5e17e69fd4-kube-api-access-ktdr7\") pod \"e41af8fc-47f1-428a-ad7b-6e5e17e69fd4\" (UID: \"e41af8fc-47f1-428a-ad7b-6e5e17e69fd4\") " Jul 24 16:34:17.381138 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:17.381117 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41af8fc-47f1-428a-ad7b-6e5e17e69fd4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e41af8fc-47f1-428a-ad7b-6e5e17e69fd4" (UID: "e41af8fc-47f1-428a-ad7b-6e5e17e69fd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:34:17.381271 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:17.381251 2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e41af8fc-47f1-428a-ad7b-6e5e17e69fd4-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:34:17.383068 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:17.383046 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41af8fc-47f1-428a-ad7b-6e5e17e69fd4-kube-api-access-ktdr7" (OuterVolumeSpecName: "kube-api-access-ktdr7") pod "e41af8fc-47f1-428a-ad7b-6e5e17e69fd4" (UID: "e41af8fc-47f1-428a-ad7b-6e5e17e69fd4"). InnerVolumeSpecName "kube-api-access-ktdr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:34:17.482130 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:17.482094 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktdr7\" (UniqueName: \"kubernetes.io/projected/e41af8fc-47f1-428a-ad7b-6e5e17e69fd4-kube-api-access-ktdr7\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:34:17.884534 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:17.884499 2569 generic.go:356] "Generic (PLEG): container finished" podID="e41af8fc-47f1-428a-ad7b-6e5e17e69fd4" containerID="f23150b4eb1464a7593428aa3155d344e939af4f409c602b840b3e6900d81216" exitCode=0 Jul 24 16:34:17.884700 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:17.884540 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd" Jul 24 16:34:17.884700 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:17.884579 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd" event={"ID":"e41af8fc-47f1-428a-ad7b-6e5e17e69fd4","Type":"ContainerDied","Data":"f23150b4eb1464a7593428aa3155d344e939af4f409c602b840b3e6900d81216"} Jul 24 16:34:17.884700 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:17.884630 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd" event={"ID":"e41af8fc-47f1-428a-ad7b-6e5e17e69fd4","Type":"ContainerDied","Data":"66a53a81e8a59dd8c05b9679b9769a25b572f3da8beb8268538c565e992a5d2c"} Jul 24 16:34:17.884700 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:17.884655 2569 scope.go:117] "RemoveContainer" containerID="f23150b4eb1464a7593428aa3155d344e939af4f409c602b840b3e6900d81216" Jul 24 16:34:17.896132 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:17.896116 2569 scope.go:117] "RemoveContainer" containerID="f23150b4eb1464a7593428aa3155d344e939af4f409c602b840b3e6900d81216" Jul 24 16:34:17.896492 ip-10-0-141-58 kubenswrapper[2569]: E0724 16:34:17.896470 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f23150b4eb1464a7593428aa3155d344e939af4f409c602b840b3e6900d81216\": container with ID starting with f23150b4eb1464a7593428aa3155d344e939af4f409c602b840b3e6900d81216 not found: ID does not exist" containerID="f23150b4eb1464a7593428aa3155d344e939af4f409c602b840b3e6900d81216" Jul 24 16:34:17.896581 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:17.896504 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f23150b4eb1464a7593428aa3155d344e939af4f409c602b840b3e6900d81216"} err="failed to get container status \"f23150b4eb1464a7593428aa3155d344e939af4f409c602b840b3e6900d81216\": rpc error: code = NotFound desc = could not find container \"f23150b4eb1464a7593428aa3155d344e939af4f409c602b840b3e6900d81216\": container with ID starting with f23150b4eb1464a7593428aa3155d344e939af4f409c602b840b3e6900d81216 not found: ID does not exist" Jul 24 16:34:17.907100 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:17.907037 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd"] Jul 24 16:34:17.911639 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:17.911616 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2ba900-5b7b658f44-zxfcd"] Jul 24 16:34:17.961388 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:17.961350 2569 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Jul 24 16:34:17.961558 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:17.961405 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" podUID="ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Jul 24 16:34:18.961487 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:18.961445 2569 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Jul 24 16:34:18.961905 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:18.961509 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" podUID="ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Jul 24 16:34:19.629603 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:19.629569 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e41af8fc-47f1-428a-ad7b-6e5e17e69fd4" path="/var/lib/kubelet/pods/e41af8fc-47f1-428a-ad7b-6e5e17e69fd4/volumes" Jul 24 16:34:19.686527 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:19.686500 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:34:19.965051 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:19.965025 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:34:19.966106 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:19.966087 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:34:27.762390 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:27.762354 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg"] Jul 24 16:34:27.762875 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:27.762566 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" podUID="ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" containerName="istio-proxy" containerID="cri-o://83481c8064a69fc8a634f8f2fa3ae5488f3d8f7a1a6a6a03ebc8ce91e1cf31a0" gracePeriod=30 Jul 24 16:34:32.986438 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:32.986400 2569 generic.go:356] "Generic (PLEG): container finished" podID="ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" containerID="83481c8064a69fc8a634f8f2fa3ae5488f3d8f7a1a6a6a03ebc8ce91e1cf31a0" exitCode=0 Jul 24 16:34:32.986902 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:32.986466 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" event={"ID":"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c","Type":"ContainerDied","Data":"83481c8064a69fc8a634f8f2fa3ae5488f3d8f7a1a6a6a03ebc8ce91e1cf31a0"} Jul 24 16:34:32.986902 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:32.986532 2569 scope.go:117] "RemoveContainer" containerID="06c07dc8621a4a35ce5ed18416ba9516286075c1ba2df0900e2fdb66274678d2" Jul 24 16:34:33.045356 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.045331 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:34:33.140870 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.140838 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dxnl\" (UniqueName: \"kubernetes.io/projected/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-kube-api-access-9dxnl\") pod \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " Jul 24 16:34:33.141062 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.140899 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-envoy\") pod \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " Jul 24 16:34:33.141062 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.140945 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-workload-socket\") pod \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " Jul 24 16:34:33.141062 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.140971 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-data\") pod \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " Jul 24 16:34:33.141062 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.140998 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istiod-ca-cert\") pod \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " Jul 24 16:34:33.141062 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.141038 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-workload-certs\") pod \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " Jul 24 16:34:33.141418 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.141075 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-podinfo\") pod \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " Jul 24 16:34:33.141418 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.141114 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-credential-socket\") pod \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " Jul 24 16:34:33.141418 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.141153 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-token\") pod \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\" (UID: \"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c\") " Jul 24 16:34:33.141418 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.141317 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" (UID: "ca37a3b3-b985-4e77-a9b2-009e94ca4d3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:34:33.141635 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.141553 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" (UID: "ca37a3b3-b985-4e77-a9b2-009e94ca4d3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:34:33.141712 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.141687 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-workload-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:34:33.141948 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.141802 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istiod-ca-cert\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:34:33.141948 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.141755 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" (UID: "ca37a3b3-b985-4e77-a9b2-009e94ca4d3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:34:33.141948 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.141921 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" (UID: "ca37a3b3-b985-4e77-a9b2-009e94ca4d3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:34:33.142447 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.142416 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-data" (OuterVolumeSpecName: "istio-data") pod "ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" (UID: "ca37a3b3-b985-4e77-a9b2-009e94ca4d3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:34:33.143381 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.143363 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" (UID: "ca37a3b3-b985-4e77-a9b2-009e94ca4d3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:34:33.143667 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.143650 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-kube-api-access-9dxnl" (OuterVolumeSpecName: "kube-api-access-9dxnl") pod "ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" (UID: "ca37a3b3-b985-4e77-a9b2-009e94ca4d3c"). InnerVolumeSpecName "kube-api-access-9dxnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:34:33.143848 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.143822 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" (UID: "ca37a3b3-b985-4e77-a9b2-009e94ca4d3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 16:34:33.143954 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.143847 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-token" (OuterVolumeSpecName: "istio-token") pod "ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" (UID: "ca37a3b3-b985-4e77-a9b2-009e94ca4d3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:34:33.242706 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.242674 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dxnl\" (UniqueName: \"kubernetes.io/projected/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-kube-api-access-9dxnl\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:34:33.242706 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.242701 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-envoy\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:34:33.242706 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.242713 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-data\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:34:33.242934 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.242721 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-workload-certs\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:34:33.242934 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.242729 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-podinfo\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:34:33.242934 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.242737 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-credential-socket\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:34:33.242934 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.242745 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c-istio-token\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:34:33.994338 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.994253 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" Jul 24 16:34:33.994338 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.994290 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg" event={"ID":"ca37a3b3-b985-4e77-a9b2-009e94ca4d3c","Type":"ContainerDied","Data":"656c207dcf152b51cd1edf23c6a451e748e4c83ec33501f05efb4a92f60c5928"} Jul 24 16:34:33.994806 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:33.994353 2569 scope.go:117] "RemoveContainer" containerID="83481c8064a69fc8a634f8f2fa3ae5488f3d8f7a1a6a6a03ebc8ce91e1cf31a0" Jul 24 16:34:34.017038 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:34.017007 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg"] Jul 24 16:34:34.022040 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:34.022015 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2ba900-openshift-default-54474d75b-lvbvg"] Jul 24 16:34:35.631119 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:35.631068 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" path="/var/lib/kubelet/pods/ca37a3b3-b985-4e77-a9b2-009e94ca4d3c/volumes" Jul 24 16:34:47.387820 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:47.387735 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg"] Jul 24 16:34:47.388405 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:47.387999 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" podUID="9924f396-4ec9-41df-8ade-6957abfb3287" containerName="maas-api" containerID="cri-o://c0f2ccbadb6df73a2fc81be51865c8f3c93aa42037327de31cd90741e1528fec" gracePeriod=30 Jul 24 16:34:47.665523 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:47.665499 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" Jul 24 16:34:47.786802 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:47.786766 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9924f396-4ec9-41df-8ade-6957abfb3287-maas-api-tls\") pod \"9924f396-4ec9-41df-8ade-6957abfb3287\" (UID: \"9924f396-4ec9-41df-8ade-6957abfb3287\") " Jul 24 16:34:47.786977 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:47.786885 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9924f396-4ec9-41df-8ade-6957abfb3287-openshift-service-ca\") pod \"9924f396-4ec9-41df-8ade-6957abfb3287\" (UID: \"9924f396-4ec9-41df-8ade-6957abfb3287\") " Jul 24 16:34:47.786977 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:47.786928 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6zt2\" (UniqueName: \"kubernetes.io/projected/9924f396-4ec9-41df-8ade-6957abfb3287-kube-api-access-w6zt2\") pod \"9924f396-4ec9-41df-8ade-6957abfb3287\" (UID: \"9924f396-4ec9-41df-8ade-6957abfb3287\") " Jul 24 16:34:47.787293 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:47.787270 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9924f396-4ec9-41df-8ade-6957abfb3287-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9924f396-4ec9-41df-8ade-6957abfb3287" (UID: "9924f396-4ec9-41df-8ade-6957abfb3287"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:34:47.789032 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:47.789010 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9924f396-4ec9-41df-8ade-6957abfb3287-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9924f396-4ec9-41df-8ade-6957abfb3287" (UID: "9924f396-4ec9-41df-8ade-6957abfb3287"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:34:47.789109 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:47.789032 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9924f396-4ec9-41df-8ade-6957abfb3287-kube-api-access-w6zt2" (OuterVolumeSpecName: "kube-api-access-w6zt2") pod "9924f396-4ec9-41df-8ade-6957abfb3287" (UID: "9924f396-4ec9-41df-8ade-6957abfb3287"). InnerVolumeSpecName "kube-api-access-w6zt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:34:47.888440 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:47.888402 2569 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9924f396-4ec9-41df-8ade-6957abfb3287-openshift-service-ca\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:34:47.888440 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:47.888433 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6zt2\" (UniqueName: \"kubernetes.io/projected/9924f396-4ec9-41df-8ade-6957abfb3287-kube-api-access-w6zt2\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:34:47.888440 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:47.888443 2569 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9924f396-4ec9-41df-8ade-6957abfb3287-maas-api-tls\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:34:48.088654 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:48.088565 2569 generic.go:356] "Generic (PLEG): container finished" podID="9924f396-4ec9-41df-8ade-6957abfb3287" containerID="c0f2ccbadb6df73a2fc81be51865c8f3c93aa42037327de31cd90741e1528fec" exitCode=0 Jul 24 16:34:48.088654 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:48.088614 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" Jul 24 16:34:48.088913 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:48.088652 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" event={"ID":"9924f396-4ec9-41df-8ade-6957abfb3287","Type":"ContainerDied","Data":"c0f2ccbadb6df73a2fc81be51865c8f3c93aa42037327de31cd90741e1528fec"} Jul 24 16:34:48.088913 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:48.088716 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg" event={"ID":"9924f396-4ec9-41df-8ade-6957abfb3287","Type":"ContainerDied","Data":"db4af39e58aff57f3ba6390f62c6fa984f383e714b180a5ccc2fee20230de143"} Jul 24 16:34:48.088913 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:48.088745 2569 scope.go:117] "RemoveContainer" containerID="c0f2ccbadb6df73a2fc81be51865c8f3c93aa42037327de31cd90741e1528fec" Jul 24 16:34:48.099050 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:48.099033 2569 scope.go:117] "RemoveContainer" containerID="c0f2ccbadb6df73a2fc81be51865c8f3c93aa42037327de31cd90741e1528fec" Jul 24 16:34:48.099388 ip-10-0-141-58 kubenswrapper[2569]: E0724 16:34:48.099371 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f2ccbadb6df73a2fc81be51865c8f3c93aa42037327de31cd90741e1528fec\": container with ID starting with c0f2ccbadb6df73a2fc81be51865c8f3c93aa42037327de31cd90741e1528fec not found: ID does not exist" containerID="c0f2ccbadb6df73a2fc81be51865c8f3c93aa42037327de31cd90741e1528fec" Jul 24 16:34:48.099434 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:48.099396 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f2ccbadb6df73a2fc81be51865c8f3c93aa42037327de31cd90741e1528fec"} err="failed to get container status \"c0f2ccbadb6df73a2fc81be51865c8f3c93aa42037327de31cd90741e1528fec\": rpc error: code = NotFound desc = could not find container \"c0f2ccbadb6df73a2fc81be51865c8f3c93aa42037327de31cd90741e1528fec\": container with ID starting with c0f2ccbadb6df73a2fc81be51865c8f3c93aa42037327de31cd90741e1528fec not found: ID does not exist" Jul 24 16:34:48.112016 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:48.111992 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg"] Jul 24 16:34:48.115656 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:48.115634 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-0107ee-cf5959dbd-n7qfg"] Jul 24 16:34:49.629639 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:34:49.629600 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9924f396-4ec9-41df-8ade-6957abfb3287" path="/var/lib/kubelet/pods/9924f396-4ec9-41df-8ade-6957abfb3287/volumes" Jul 24 16:35:25.858954 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:25.858924 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 16:35:25.864975 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:25.864946 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 16:35:25.865198 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:25.865172 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 16:35:25.871170 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:25.871150 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qm2n9_fc74e7b2-f0e1-4fe3-b02a-567e2f726143/ovn-acl-logging/0.log" Jul 24 16:35:56.790665 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:56.790627 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd"] Jul 24 16:35:56.791288 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:56.790897 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd" podUID="150a0264-5b33-49c3-ba4b-4e6bae786f86" containerName="payload-processing" containerID="cri-o://c0400a002fffb6451b2b9c63bf405a38a9c7224fc8bcb1437634a62283baa65c" gracePeriod=30 Jul 24 16:35:57.060280 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:57.060259 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd" Jul 24 16:35:57.070694 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:57.070666 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsnhf\" (UniqueName: \"kubernetes.io/projected/150a0264-5b33-49c3-ba4b-4e6bae786f86-kube-api-access-gsnhf\") pod \"150a0264-5b33-49c3-ba4b-4e6bae786f86\" (UID: \"150a0264-5b33-49c3-ba4b-4e6bae786f86\") " Jul 24 16:35:57.070811 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:57.070766 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150a0264-5b33-49c3-ba4b-4e6bae786f86-plugins-config-volume\") pod \"150a0264-5b33-49c3-ba4b-4e6bae786f86\" (UID: \"150a0264-5b33-49c3-ba4b-4e6bae786f86\") " Jul 24 16:35:57.071110 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:57.071081 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150a0264-5b33-49c3-ba4b-4e6bae786f86-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "150a0264-5b33-49c3-ba4b-4e6bae786f86" (UID: "150a0264-5b33-49c3-ba4b-4e6bae786f86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:35:57.073078 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:57.073050 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150a0264-5b33-49c3-ba4b-4e6bae786f86-kube-api-access-gsnhf" (OuterVolumeSpecName: "kube-api-access-gsnhf") pod "150a0264-5b33-49c3-ba4b-4e6bae786f86" (UID: "150a0264-5b33-49c3-ba4b-4e6bae786f86"). InnerVolumeSpecName "kube-api-access-gsnhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:35:57.172354 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:57.172315 2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150a0264-5b33-49c3-ba4b-4e6bae786f86-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:35:57.172354 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:57.172346 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsnhf\" (UniqueName: \"kubernetes.io/projected/150a0264-5b33-49c3-ba4b-4e6bae786f86-kube-api-access-gsnhf\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:35:57.537965 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:57.537929 2569 generic.go:356] "Generic (PLEG): container finished" podID="150a0264-5b33-49c3-ba4b-4e6bae786f86" containerID="c0400a002fffb6451b2b9c63bf405a38a9c7224fc8bcb1437634a62283baa65c" exitCode=0 Jul 24 16:35:57.538133 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:57.537989 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd" Jul 24 16:35:57.538133 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:57.538011 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd" event={"ID":"150a0264-5b33-49c3-ba4b-4e6bae786f86","Type":"ContainerDied","Data":"c0400a002fffb6451b2b9c63bf405a38a9c7224fc8bcb1437634a62283baa65c"} Jul 24 16:35:57.538133 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:57.538090 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd" event={"ID":"150a0264-5b33-49c3-ba4b-4e6bae786f86","Type":"ContainerDied","Data":"fe4f309f9eb7eed1e2ccfb00d579be988e53d07b04b063fcd54cad36d2ac716d"} Jul 24 16:35:57.538133 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:57.538113 2569 scope.go:117] "RemoveContainer" containerID="c0400a002fffb6451b2b9c63bf405a38a9c7224fc8bcb1437634a62283baa65c" Jul 24 16:35:57.548148 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:57.548129 2569 scope.go:117] "RemoveContainer" containerID="c0400a002fffb6451b2b9c63bf405a38a9c7224fc8bcb1437634a62283baa65c" Jul 24 16:35:57.548435 ip-10-0-141-58 kubenswrapper[2569]: E0724 16:35:57.548413 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0400a002fffb6451b2b9c63bf405a38a9c7224fc8bcb1437634a62283baa65c\": container with ID starting with c0400a002fffb6451b2b9c63bf405a38a9c7224fc8bcb1437634a62283baa65c not found: ID does not exist" containerID="c0400a002fffb6451b2b9c63bf405a38a9c7224fc8bcb1437634a62283baa65c" Jul 24 16:35:57.548526 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:57.548440 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0400a002fffb6451b2b9c63bf405a38a9c7224fc8bcb1437634a62283baa65c"} err="failed to get container status \"c0400a002fffb6451b2b9c63bf405a38a9c7224fc8bcb1437634a62283baa65c\": rpc error: code = NotFound desc = could not find container \"c0400a002fffb6451b2b9c63bf405a38a9c7224fc8bcb1437634a62283baa65c\": container with ID starting with c0400a002fffb6451b2b9c63bf405a38a9c7224fc8bcb1437634a62283baa65c not found: ID does not exist" Jul 24 16:35:57.564040 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:57.564017 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd"] Jul 24 16:35:57.566613 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:57.566598 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-52d140-685b875984-5ftcd"] Jul 24 16:35:57.629927 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:35:57.629892 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="150a0264-5b33-49c3-ba4b-4e6bae786f86" path="/var/lib/kubelet/pods/150a0264-5b33-49c3-ba4b-4e6bae786f86/volumes" Jul 24 16:36:41.158611 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:41.158572 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25"] Jul 24 16:36:41.159374 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:41.158835 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25" podUID="981696de-6034-44af-b650-efa0ddfc2788" containerName="payload-processing" containerID="cri-o://c6a84d72d940dd08b3926e458ec3337b20e4567b513b526332a10b4a0da3e866" gracePeriod=30 Jul 24 16:36:41.433563 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:41.433537 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25" Jul 24 16:36:41.499641 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:41.499610 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981696de-6034-44af-b650-efa0ddfc2788-plugins-config-volume\") pod \"981696de-6034-44af-b650-efa0ddfc2788\" (UID: \"981696de-6034-44af-b650-efa0ddfc2788\") " Jul 24 16:36:41.499833 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:41.499666 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnfbg\" (UniqueName: \"kubernetes.io/projected/981696de-6034-44af-b650-efa0ddfc2788-kube-api-access-lnfbg\") pod \"981696de-6034-44af-b650-efa0ddfc2788\" (UID: \"981696de-6034-44af-b650-efa0ddfc2788\") " Jul 24 16:36:41.500046 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:41.500018 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981696de-6034-44af-b650-efa0ddfc2788-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "981696de-6034-44af-b650-efa0ddfc2788" (UID: "981696de-6034-44af-b650-efa0ddfc2788"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:36:41.501810 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:41.501782 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981696de-6034-44af-b650-efa0ddfc2788-kube-api-access-lnfbg" (OuterVolumeSpecName: "kube-api-access-lnfbg") pod "981696de-6034-44af-b650-efa0ddfc2788" (UID: "981696de-6034-44af-b650-efa0ddfc2788"). InnerVolumeSpecName "kube-api-access-lnfbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:36:41.601084 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:41.601043 2569 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981696de-6034-44af-b650-efa0ddfc2788-plugins-config-volume\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:36:41.601084 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:41.601076 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnfbg\" (UniqueName: \"kubernetes.io/projected/981696de-6034-44af-b650-efa0ddfc2788-kube-api-access-lnfbg\") on node \"ip-10-0-141-58.ec2.internal\" DevicePath \"\"" Jul 24 16:36:41.817339 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:41.817241 2569 generic.go:356] "Generic (PLEG): container finished" podID="981696de-6034-44af-b650-efa0ddfc2788" containerID="c6a84d72d940dd08b3926e458ec3337b20e4567b513b526332a10b4a0da3e866" exitCode=0 Jul 24 16:36:41.817339 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:41.817294 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25" Jul 24 16:36:41.817525 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:41.817334 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25" event={"ID":"981696de-6034-44af-b650-efa0ddfc2788","Type":"ContainerDied","Data":"c6a84d72d940dd08b3926e458ec3337b20e4567b513b526332a10b4a0da3e866"} Jul 24 16:36:41.817525 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:41.817393 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25" event={"ID":"981696de-6034-44af-b650-efa0ddfc2788","Type":"ContainerDied","Data":"3ae6d52736e8c0c20057aea7c57ade46ed20f0dd5f92b8bec7a236622caae932"} Jul 24 16:36:41.817525 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:41.817416 2569 scope.go:117] "RemoveContainer" containerID="c6a84d72d940dd08b3926e458ec3337b20e4567b513b526332a10b4a0da3e866" Jul 24 16:36:41.827611 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:41.827590 2569 scope.go:117] "RemoveContainer" containerID="c6a84d72d940dd08b3926e458ec3337b20e4567b513b526332a10b4a0da3e866" Jul 24 16:36:41.827926 ip-10-0-141-58 kubenswrapper[2569]: E0724 16:36:41.827889 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a84d72d940dd08b3926e458ec3337b20e4567b513b526332a10b4a0da3e866\": container with ID starting with c6a84d72d940dd08b3926e458ec3337b20e4567b513b526332a10b4a0da3e866 not found: ID does not exist" containerID="c6a84d72d940dd08b3926e458ec3337b20e4567b513b526332a10b4a0da3e866" Jul 24 16:36:41.828015 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:41.827938 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a84d72d940dd08b3926e458ec3337b20e4567b513b526332a10b4a0da3e866"} err="failed to get container status \"c6a84d72d940dd08b3926e458ec3337b20e4567b513b526332a10b4a0da3e866\": rpc error: code = NotFound desc = could not find container \"c6a84d72d940dd08b3926e458ec3337b20e4567b513b526332a10b4a0da3e866\": container with ID starting with c6a84d72d940dd08b3926e458ec3337b20e4567b513b526332a10b4a0da3e866 not found: ID does not exist" Jul 24 16:36:41.836460 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:41.836436 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25"] Jul 24 16:36:41.841973 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:41.841948 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-82adcb-785755576b-dvb25"] Jul 24 16:36:43.629590 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:43.629557 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="981696de-6034-44af-b650-efa0ddfc2788" path="/var/lib/kubelet/pods/981696de-6034-44af-b650-efa0ddfc2788/volumes" Jul 24 16:36:52.947880 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:52.947847 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78694cc8bf-m8l45_e5c23128-5563-4307-ae00-a054c6a39f1c/authorino/0.log" Jul 24 16:36:58.507183 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:58.507152 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-jcgj8_4a0df429-8f7b-45ea-b167-96c39c5db85d/manager/0.log" Jul 24 16:36:58.636173 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:58.636146 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-qqczl_0134774e-c656-4108-99a4-57714cc9e9f3/manager/0.log" Jul 24 16:36:58.768839 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:58.768753 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-685b879f45-qkclw_2e44bd61-85ea-4dc4-b1e1-b33e685d7b9a/manager/0.log" Jul 24 16:36:59.041310 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:59.041205 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-kp9qs_c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f/manager/0.log" Jul 24 16:36:59.412538 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:59.412515 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d995667b8-fbtbs_68181f3b-f19f-459f-b325-4bedfcd80b77/copy-manifests/0.log" Jul 24 16:36:59.430830 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:59.430804 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d995667b8-fbtbs_68181f3b-f19f-459f-b325-4bedfcd80b77/manager/0.log" Jul 24 16:36:59.805602 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:36:59.805519 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-864cf74d58-9nc7m_611d9d18-c8e6-4cde-bbeb-559a55bea8cb/maas-api/0.log" Jul 24 16:37:00.284744 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:00.284687 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-wwgfx_e71ea927-92d1-4cc4-9399-9c0a2148a8c6/postgres/0.log" Jul 24 16:37:01.108385 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:01.108358 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s_373c13c5-cd6e-4565-b34e-768266c0ec26/util/0.log" Jul 24 16:37:01.115525 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:01.115500 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s_373c13c5-cd6e-4565-b34e-768266c0ec26/pull/0.log" Jul 24 16:37:01.122187 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:01.122161 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s_373c13c5-cd6e-4565-b34e-768266c0ec26/extract/0.log" Jul 24 16:37:01.249773 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:01.249744 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b_01c8ce33-7591-4b16-a830-114abddd3507/util/0.log" Jul 24 16:37:01.256915 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:01.256886 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b_01c8ce33-7591-4b16-a830-114abddd3507/pull/0.log" Jul 24 16:37:01.264199 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:01.264177 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b_01c8ce33-7591-4b16-a830-114abddd3507/extract/0.log" Jul 24 16:37:01.382032 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:01.381945 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw_c74a98b2-dd57-4428-9518-c04852f7aca8/util/0.log" Jul 24 16:37:01.389072 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:01.389050 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw_c74a98b2-dd57-4428-9518-c04852f7aca8/pull/0.log" Jul 24 16:37:01.395575 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:01.395557 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw_c74a98b2-dd57-4428-9518-c04852f7aca8/extract/0.log" Jul 24 16:37:01.513938 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:01.513909 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd_42a19dbb-d098-4980-930b-f9440ecca8ae/util/0.log" Jul 24 16:37:01.520467 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:01.520443 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd_42a19dbb-d098-4980-930b-f9440ecca8ae/pull/0.log" Jul 24 16:37:01.528903 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:01.528883 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd_42a19dbb-d098-4980-930b-f9440ecca8ae/extract/0.log" Jul 24 16:37:01.658673 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:01.658643 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78694cc8bf-m8l45_e5c23128-5563-4307-ae00-a054c6a39f1c/authorino/0.log" Jul 24 16:37:01.781643 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:01.781608 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7d8hj_6c8cef0e-76c4-41aa-a26d-eaa6c2963b4c/manager/0.log" Jul 24 16:37:01.900609 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:01.900583 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-65kzp_3db27a28-ad12-4c20-b8f6-d83a6ec42a09/manager/0.log" Jul 24 16:37:02.018681 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:02.018604 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-lx98m_a29bc5d4-1424-462d-bd48-9c5f74b7c812/kuadrant-console-plugin/0.log" Jul 24 16:37:02.154131 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:02.154089 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-8dmw4_b0dafc08-22ce-4c05-9687-95dd771ea52c/registry-server/0.log" Jul 24 16:37:02.287842 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:02.287760 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-v9kzl_b750b5de-3604-4e92-ac6f-12c31cf8d7fd/manager/0.log" Jul 24 16:37:02.552617 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:02.552536 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-cbscc_39800830-d9ce-48cb-8906-b991efdf13ff/manager/0.log" Jul 24 16:37:03.299129 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:03.299096 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-767449f4cf-hgvr5_319718c8-0b89-4ecf-ae6b-03e13f89c9fb/kube-auth-proxy/0.log" Jul 24 16:37:03.697813 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:03.697778 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-kbzx2_ca5f962c-6f6a-4360-874c-cadf65694353/payload-processing/0.log" Jul 24 16:37:03.818718 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:03.818693 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-65b698c97-9vtdf_fe32eca6-90f7-4d13-a58f-156b373a18f9/router/0.log" Jul 24 16:37:04.309779 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:04.309750 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-4plv2_13521c7f-020f-489d-b1dc-7996d725c55b/main/0.log" Jul 24 16:37:04.430113 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:04.430085 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-jqdrk_d0917d03-ecd9-4ca9-ae85-edf96a32b40b/main/0.log" Jul 24 16:37:13.560434 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:13.560403 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lj6pt_d1348e8a-07ee-4ce2-afa4-2d580edd8df2/global-pull-secret-syncer/0.log" Jul 24 16:37:13.667809 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:13.667775 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g2vqr_357d493f-b503-4d18-b37c-e8e7168f84c2/konnectivity-agent/0.log" Jul 24 16:37:13.795063 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:13.795030 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-58.ec2.internal_7dc55cd8cdabaf6971bfe9f07f04cbd9/haproxy/0.log" Jul 24 16:37:18.157045 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:18.157017 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s_373c13c5-cd6e-4565-b34e-768266c0ec26/extract/0.log" Jul 24 16:37:18.181009 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:18.180983 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s_373c13c5-cd6e-4565-b34e-768266c0ec26/util/0.log" Jul 24 16:37:18.206081 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:18.206048 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759v5s2s_373c13c5-cd6e-4565-b34e-768266c0ec26/pull/0.log" Jul 24 16:37:18.234255 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:18.234198 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b_01c8ce33-7591-4b16-a830-114abddd3507/extract/0.log" Jul 24 16:37:18.255755 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:18.255730 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b_01c8ce33-7591-4b16-a830-114abddd3507/util/0.log" Jul 24 16:37:18.277264 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:18.277233 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0hhl9b_01c8ce33-7591-4b16-a830-114abddd3507/pull/0.log" Jul 24 16:37:18.307731 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:18.307701 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw_c74a98b2-dd57-4428-9518-c04852f7aca8/extract/0.log" Jul 24 16:37:18.332815 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:18.332729 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw_c74a98b2-dd57-4428-9518-c04852f7aca8/util/0.log" Jul 24 16:37:18.355266 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:18.355239 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7377klw_c74a98b2-dd57-4428-9518-c04852f7aca8/pull/0.log" Jul 24 16:37:18.386629 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:18.386591 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd_42a19dbb-d098-4980-930b-f9440ecca8ae/extract/0.log" Jul 24 16:37:18.407788 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:18.407758 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd_42a19dbb-d098-4980-930b-f9440ecca8ae/util/0.log" Jul 24 16:37:18.432069 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:18.432042 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef169jfd_42a19dbb-d098-4980-930b-f9440ecca8ae/pull/0.log" Jul 24 16:37:19.087761 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:19.087731 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78694cc8bf-m8l45_e5c23128-5563-4307-ae00-a054c6a39f1c/authorino/0.log" Jul 24 16:37:19.123421 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:19.123376 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7d8hj_6c8cef0e-76c4-41aa-a26d-eaa6c2963b4c/manager/0.log" Jul 24 16:37:19.156826 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:19.156798 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-65kzp_3db27a28-ad12-4c20-b8f6-d83a6ec42a09/manager/0.log" Jul 24 16:37:19.182243 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:19.182205 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-lx98m_a29bc5d4-1424-462d-bd48-9c5f74b7c812/kuadrant-console-plugin/0.log" Jul 24 16:37:19.237332 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:19.237296 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-8dmw4_b0dafc08-22ce-4c05-9687-95dd771ea52c/registry-server/0.log" Jul 24 16:37:19.446792 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:19.446762 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-v9kzl_b750b5de-3604-4e92-ac6f-12c31cf8d7fd/manager/0.log" Jul 24 16:37:19.577560 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:19.577528 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-cbscc_39800830-d9ce-48cb-8906-b991efdf13ff/manager/0.log" Jul 24 16:37:21.282173 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:21.282144 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9df211ef-c1bc-4f57-9959-8ba32bc884d5/alertmanager/0.log" Jul 24 16:37:21.306340 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:21.306303 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9df211ef-c1bc-4f57-9959-8ba32bc884d5/config-reloader/0.log" Jul 24 16:37:21.327380 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:21.327350 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9df211ef-c1bc-4f57-9959-8ba32bc884d5/kube-rbac-proxy-web/0.log" Jul 24 16:37:21.353349 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:21.353323 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9df211ef-c1bc-4f57-9959-8ba32bc884d5/kube-rbac-proxy/0.log" Jul 24 16:37:21.373338 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:21.373293 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9df211ef-c1bc-4f57-9959-8ba32bc884d5/kube-rbac-proxy-metric/0.log" Jul 24 16:37:21.399125 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:21.399098 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9df211ef-c1bc-4f57-9959-8ba32bc884d5/prom-label-proxy/0.log" Jul 24 16:37:21.425288 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:21.425262 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9df211ef-c1bc-4f57-9959-8ba32bc884d5/init-config-reloader/0.log" Jul 24 16:37:21.477479 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:21.477443 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jp9sz_61cb38d8-9f5e-495b-931b-7561758ce3a3/cluster-monitoring-operator/0.log" Jul 24 16:37:21.606634 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:21.606535 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-vhfgv_cb84e7d4-45f5-490c-9ca2-a30a89a05a1e/monitoring-plugin/0.log" Jul 24 16:37:21.776876 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:21.776849 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mfwls_8fbcb1ed-20ae-45aa-9406-141c2f9fddd0/node-exporter/0.log" Jul 24 16:37:21.798981 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:21.798948 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mfwls_8fbcb1ed-20ae-45aa-9406-141c2f9fddd0/kube-rbac-proxy/0.log" Jul 24 16:37:21.821678 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:21.821651 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mfwls_8fbcb1ed-20ae-45aa-9406-141c2f9fddd0/init-textfile/0.log" Jul 24 16:37:21.854507 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:21.854483 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-2k59p_4092a80d-7a41-487d-a985-8f8bee7528f0/kube-rbac-proxy-main/0.log" Jul 24 16:37:21.875388 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:21.875302 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-2k59p_4092a80d-7a41-487d-a985-8f8bee7528f0/kube-rbac-proxy-self/0.log" Jul 24 16:37:21.896901 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:21.896872 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-2k59p_4092a80d-7a41-487d-a985-8f8bee7528f0/openshift-state-metrics/0.log" Jul 24 16:37:22.165124 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:22.165088 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d7966675b-28qzj_e62d39b7-bb9b-46f9-9106-eb880de55932/telemeter-client/0.log" Jul 24 16:37:22.189165 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:22.189107 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d7966675b-28qzj_e62d39b7-bb9b-46f9-9106-eb880de55932/reload/0.log" Jul 24 16:37:22.210225 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:22.210185 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-d7966675b-28qzj_e62d39b7-bb9b-46f9-9106-eb880de55932/kube-rbac-proxy/0.log" Jul 24 16:37:22.243004 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:22.242964 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759b6bd54c-dxnbh_e99f3198-7006-4f15-878b-9329f27bda15/thanos-query/0.log" Jul 24 16:37:22.265675 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:22.265650 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759b6bd54c-dxnbh_e99f3198-7006-4f15-878b-9329f27bda15/kube-rbac-proxy-web/0.log" Jul 24 16:37:22.286804 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:22.286773 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759b6bd54c-dxnbh_e99f3198-7006-4f15-878b-9329f27bda15/kube-rbac-proxy/0.log" Jul 24 16:37:22.309737 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:22.309712 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759b6bd54c-dxnbh_e99f3198-7006-4f15-878b-9329f27bda15/prom-label-proxy/0.log" Jul 24 16:37:22.329705 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:22.329676 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759b6bd54c-dxnbh_e99f3198-7006-4f15-878b-9329f27bda15/kube-rbac-proxy-rules/0.log" Jul 24 16:37:22.361028 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:22.360998 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-759b6bd54c-dxnbh_e99f3198-7006-4f15-878b-9329f27bda15/kube-rbac-proxy-metrics/0.log" Jul 24 16:37:23.470267 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.470205 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-d44kg_2f97027f-5262-48c0-acd9-61ccc7b95bbf/networking-console-plugin/0.log" Jul 24 16:37:23.815549 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.815465 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c"] Jul 24 16:37:23.815952 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.815939 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" containerName="istio-proxy" Jul 24 16:37:23.815996 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.815953 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" containerName="istio-proxy" Jul 24 16:37:23.815996 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.815963 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e41af8fc-47f1-428a-ad7b-6e5e17e69fd4" containerName="payload-processing" Jul 24 16:37:23.815996 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.815969 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e41af8fc-47f1-428a-ad7b-6e5e17e69fd4" containerName="payload-processing" Jul 24 16:37:23.815996 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.815980 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9924f396-4ec9-41df-8ade-6957abfb3287" containerName="maas-api" Jul 24 16:37:23.815996 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.815986 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9924f396-4ec9-41df-8ade-6957abfb3287" containerName="maas-api" Jul 24 16:37:23.816153 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.816002 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" containerName="istio-proxy" Jul 24 16:37:23.816153 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.816007 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" containerName="istio-proxy" Jul 24 16:37:23.816153 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.816015 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="981696de-6034-44af-b650-efa0ddfc2788" containerName="payload-processing" Jul 24 16:37:23.816153 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.816021 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="981696de-6034-44af-b650-efa0ddfc2788" containerName="payload-processing" Jul 24 16:37:23.816153 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.816028 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="150a0264-5b33-49c3-ba4b-4e6bae786f86" containerName="payload-processing" Jul 24 16:37:23.816153 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.816033 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="150a0264-5b33-49c3-ba4b-4e6bae786f86" containerName="payload-processing" Jul 24 16:37:23.816153 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.816111 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="9924f396-4ec9-41df-8ade-6957abfb3287" containerName="maas-api" Jul 24 16:37:23.816153 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.816119 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e41af8fc-47f1-428a-ad7b-6e5e17e69fd4" containerName="payload-processing" Jul 24 16:37:23.816153 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.816125 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="981696de-6034-44af-b650-efa0ddfc2788" containerName="payload-processing" Jul 24 16:37:23.816153 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.816131 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="150a0264-5b33-49c3-ba4b-4e6bae786f86" containerName="payload-processing" Jul 24 16:37:23.816153 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.816135 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" containerName="istio-proxy" Jul 24 16:37:23.816153 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.816141 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca37a3b3-b985-4e77-a9b2-009e94ca4d3c" containerName="istio-proxy" Jul 24 16:37:23.819467 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.819451 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" Jul 24 16:37:23.822447 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.822424 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f8c76\"/\"default-dockercfg-6svfq\"" Jul 24 16:37:23.822590 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.822470 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f8c76\"/\"kube-root-ca.crt\"" Jul 24 16:37:23.823329 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.823311 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f8c76\"/\"openshift-service-ca.crt\"" Jul 24 16:37:23.831553 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.831532 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c"] Jul 24 16:37:23.903601 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.903568 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/00e3e9c8-c6be-4885-ace3-99f0f8564dad-podres\") pod \"perf-node-gather-daemonset-tbk2c\" (UID: \"00e3e9c8-c6be-4885-ace3-99f0f8564dad\") " pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" Jul 24 16:37:23.903601 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.903602 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00e3e9c8-c6be-4885-ace3-99f0f8564dad-sys\") pod \"perf-node-gather-daemonset-tbk2c\" (UID: \"00e3e9c8-c6be-4885-ace3-99f0f8564dad\") " pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" Jul 24 16:37:23.903816 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.903674 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwtcc\" (UniqueName: \"kubernetes.io/projected/00e3e9c8-c6be-4885-ace3-99f0f8564dad-kube-api-access-mwtcc\") pod \"perf-node-gather-daemonset-tbk2c\" (UID: \"00e3e9c8-c6be-4885-ace3-99f0f8564dad\") " pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" Jul 24 16:37:23.903816 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.903728 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/00e3e9c8-c6be-4885-ace3-99f0f8564dad-proc\") pod \"perf-node-gather-daemonset-tbk2c\" (UID: \"00e3e9c8-c6be-4885-ace3-99f0f8564dad\") " pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" Jul 24 16:37:23.903894 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:23.903825 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00e3e9c8-c6be-4885-ace3-99f0f8564dad-lib-modules\") pod \"perf-node-gather-daemonset-tbk2c\" (UID: \"00e3e9c8-c6be-4885-ace3-99f0f8564dad\") " pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" Jul 24 16:37:24.005165 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:24.005134 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00e3e9c8-c6be-4885-ace3-99f0f8564dad-lib-modules\") pod \"perf-node-gather-daemonset-tbk2c\" (UID: \"00e3e9c8-c6be-4885-ace3-99f0f8564dad\") " pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" Jul 24 16:37:24.005394 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:24.005180 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/00e3e9c8-c6be-4885-ace3-99f0f8564dad-podres\") pod \"perf-node-gather-daemonset-tbk2c\" (UID: \"00e3e9c8-c6be-4885-ace3-99f0f8564dad\") " pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" Jul 24 16:37:24.005394 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:24.005231 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00e3e9c8-c6be-4885-ace3-99f0f8564dad-sys\") pod \"perf-node-gather-daemonset-tbk2c\" (UID: \"00e3e9c8-c6be-4885-ace3-99f0f8564dad\") " pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" Jul 24 16:37:24.005394 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:24.005293 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00e3e9c8-c6be-4885-ace3-99f0f8564dad-sys\") pod \"perf-node-gather-daemonset-tbk2c\" (UID: \"00e3e9c8-c6be-4885-ace3-99f0f8564dad\") " pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" Jul 24 16:37:24.005394 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:24.005349 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00e3e9c8-c6be-4885-ace3-99f0f8564dad-lib-modules\") pod \"perf-node-gather-daemonset-tbk2c\" (UID: \"00e3e9c8-c6be-4885-ace3-99f0f8564dad\") " pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" Jul 24 16:37:24.005394 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:24.005345 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwtcc\" (UniqueName: \"kubernetes.io/projected/00e3e9c8-c6be-4885-ace3-99f0f8564dad-kube-api-access-mwtcc\") pod \"perf-node-gather-daemonset-tbk2c\" (UID: \"00e3e9c8-c6be-4885-ace3-99f0f8564dad\") " pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" Jul 24 16:37:24.005642 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:24.005421 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/00e3e9c8-c6be-4885-ace3-99f0f8564dad-proc\") pod \"perf-node-gather-daemonset-tbk2c\" (UID: \"00e3e9c8-c6be-4885-ace3-99f0f8564dad\") " pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" Jul 24 16:37:24.005642 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:24.005348 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/00e3e9c8-c6be-4885-ace3-99f0f8564dad-podres\") pod \"perf-node-gather-daemonset-tbk2c\" (UID: \"00e3e9c8-c6be-4885-ace3-99f0f8564dad\") " pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" Jul 24 16:37:24.005642 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:24.005545 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/00e3e9c8-c6be-4885-ace3-99f0f8564dad-proc\") pod \"perf-node-gather-daemonset-tbk2c\" (UID: \"00e3e9c8-c6be-4885-ace3-99f0f8564dad\") " pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" Jul 24 16:37:24.013254 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:24.013202 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwtcc\" (UniqueName: \"kubernetes.io/projected/00e3e9c8-c6be-4885-ace3-99f0f8564dad-kube-api-access-mwtcc\") pod \"perf-node-gather-daemonset-tbk2c\" (UID: \"00e3e9c8-c6be-4885-ace3-99f0f8564dad\") " pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" Jul 24 16:37:24.130631 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:24.130540 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" Jul 24 16:37:24.135917 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:24.135893 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/2.log" Jul 24 16:37:24.141466 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:24.141437 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-285sz_70e5d390-4438-46a0-9e53-fae9badc534f/console-operator/3.log" Jul 24 16:37:24.275692 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:24.275651 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c"] Jul 24 16:37:24.276508 ip-10-0-141-58 kubenswrapper[2569]: W0724 16:37:24.276484 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod00e3e9c8_c6be_4885_ace3_99f0f8564dad.slice/crio-abc93334c72732ab31b9f096c75466ede5f807a47ac358d8a5e2843bd73ffb73 WatchSource:0}: Error finding container abc93334c72732ab31b9f096c75466ede5f807a47ac358d8a5e2843bd73ffb73: Status 404 returned error can't find the container with id abc93334c72732ab31b9f096c75466ede5f807a47ac358d8a5e2843bd73ffb73 Jul 24 16:37:24.690985 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:24.690951 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-799d79994b-xrnsx_89fad40f-43a3-4f28-ae58-840e3d6f2d7a/console/0.log" Jul 24 16:37:25.099963 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:25.099867 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" event={"ID":"00e3e9c8-c6be-4885-ace3-99f0f8564dad","Type":"ContainerStarted","Data":"7c5f23142b987988f37d03fd908e5a2eaa57034acf0e0dab296180014d8f0456"} Jul 24 16:37:25.099963 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:25.099933 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" event={"ID":"00e3e9c8-c6be-4885-ace3-99f0f8564dad","Type":"ContainerStarted","Data":"abc93334c72732ab31b9f096c75466ede5f807a47ac358d8a5e2843bd73ffb73"} Jul 24 16:37:25.100158 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:25.099984 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" Jul 24 16:37:25.115844 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:25.115797 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" podStartSLOduration=2.115782108 podStartE2EDuration="2.115782108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:37:25.115505301 +0000 UTC m=+3120.033475602" watchObservedRunningTime="2026-07-24 16:37:25.115782108 +0000 UTC m=+3120.033752407" Jul 24 16:37:25.325359 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:25.325321 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-d8p9n_c2185c30-e6e6-4623-9bcb-1a0b6baf046d/volume-data-source-validator/0.log" Jul 24 16:37:26.377104 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:26.377027 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fwkmh_20a41a97-5d30-42f9-8b18-138802f0f554/dns/0.log" Jul 24 16:37:26.422044 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:26.422015 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fwkmh_20a41a97-5d30-42f9-8b18-138802f0f554/kube-rbac-proxy/0.log" Jul 24 16:37:26.523073 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:26.523046 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bdjkj_3447e518-ab20-411f-aa9f-4918b9246385/dns-node-resolver/0.log" Jul 24 16:37:27.141683 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:27.141656 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hvtww_34e569a7-61a3-4909-865f-36a7bd286efa/node-ca/0.log" Jul 24 16:37:28.368949 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:28.368913 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-767449f4cf-hgvr5_319718c8-0b89-4ecf-ae6b-03e13f89c9fb/kube-auth-proxy/0.log" Jul 24 16:37:28.530433 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:28.530404 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-kbzx2_ca5f962c-6f6a-4360-874c-cadf65694353/payload-processing/0.log" Jul 24 16:37:28.559015 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:28.558985 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-65b698c97-9vtdf_fe32eca6-90f7-4d13-a58f-156b373a18f9/router/0.log" Jul 24 16:37:29.368781 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:29.368752 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6v2fh_d6d6c0cc-0d5f-407c-b9ea-00cf14c8076e/serve-healthcheck-canary/0.log" Jul 24 16:37:29.881223 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:29.881190 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-zf2c2_2397af02-6f7f-4fde-905b-f733e7a77c47/insights-operator/0.log" Jul 24 16:37:29.885028 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:29.885002 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-zf2c2_2397af02-6f7f-4fde-905b-f733e7a77c47/insights-operator/1.log" Jul 24 16:37:29.995137 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:29.995107 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hxpsj_0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396/kube-rbac-proxy/0.log" Jul 24 16:37:30.019383 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:30.019357 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hxpsj_0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396/exporter/0.log" Jul 24 16:37:30.042504 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:30.042456 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hxpsj_0972d7f9-5bcd-4ca5-8e8e-e23d04b7b396/extractor/0.log" Jul 24 16:37:31.117478 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:31.117452 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f8c76/perf-node-gather-daemonset-tbk2c" Jul 24 16:37:32.419267 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:32.419232 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-jcgj8_4a0df429-8f7b-45ea-b167-96c39c5db85d/manager/0.log" Jul 24 16:37:32.809330 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:32.809254 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-qqczl_0134774e-c656-4108-99a4-57714cc9e9f3/manager/0.log" Jul 24 16:37:33.048077 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:33.048042 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-685b879f45-qkclw_2e44bd61-85ea-4dc4-b1e1-b33e685d7b9a/manager/0.log" Jul 24 16:37:33.213473 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:33.213444 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-kp9qs_c5848b14-2d4b-4cee-b2d4-6ed2f1f66c5f/manager/0.log" Jul 24 16:37:33.480947 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:33.480832 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d995667b8-fbtbs_68181f3b-f19f-459f-b325-4bedfcd80b77/manager/0.log" Jul 24 16:37:33.501234 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:33.501191 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d995667b8-fbtbs_68181f3b-f19f-459f-b325-4bedfcd80b77/copy-manifests/0.log" Jul 24 16:37:34.550922 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:34.550840 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-767449f4cf-hgvr5_319718c8-0b89-4ecf-ae6b-03e13f89c9fb/kube-auth-proxy/0.log" Jul 24 16:37:34.669113 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:34.669081 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-kbzx2_ca5f962c-6f6a-4360-874c-cadf65694353/payload-processing/0.log" Jul 24 16:37:34.683699 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:34.683669 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-65b698c97-9vtdf_fe32eca6-90f7-4d13-a58f-156b373a18f9/router/0.log" Jul 24 16:37:34.944154 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:34.944121 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-s9ndj_2176d9a3-1ea1-4527-916b-0231cf449d39/openshift-lws-operator/0.log" Jul 24 16:37:36.462836 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:36.462805 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-767449f4cf-hgvr5_319718c8-0b89-4ecf-ae6b-03e13f89c9fb/kube-auth-proxy/0.log" Jul 24 16:37:36.606682 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:36.606655 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-kbzx2_ca5f962c-6f6a-4360-874c-cadf65694353/payload-processing/0.log" Jul 24 16:37:36.625121 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:36.625094 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-65b698c97-9vtdf_fe32eca6-90f7-4d13-a58f-156b373a18f9/router/0.log" Jul 24 16:37:40.398783 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:40.398720 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m_ea54c847-2195-4b03-9e9d-31efaf46de3f/kube-storage-version-migrator-operator/1.log" Jul 24 16:37:40.400445 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:40.400425 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-q9w5m_ea54c847-2195-4b03-9e9d-31efaf46de3f/kube-storage-version-migrator-operator/0.log" Jul 24 16:37:41.625919 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:41.625887 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lkwbb_6f58d4fe-4a3c-466c-8001-85fd8797a32a/kube-multus-additional-cni-plugins/0.log" Jul 24 16:37:41.646171 ip-10-0-141-58 kubenswrapper[2569]: I0724 16:37:41.646135 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lkwbb_6f58d4fe-4a3c-466c-8001-85fd8797a32a/egress-router-binary-copy/0.log"