Jul 06 18:58:56.826918 ip-10-0-141-237 systemd[1]: Starting Kubernetes Kubelet... Jul 06 18:58:57.247229 ip-10-0-141-237 kubenswrapper[2553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 18:58:57.247229 ip-10-0-141-237 kubenswrapper[2553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 18:58:57.247229 ip-10-0-141-237 kubenswrapper[2553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 18:58:57.247229 ip-10-0-141-237 kubenswrapper[2553]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 18:58:57.247229 ip-10-0-141-237 kubenswrapper[2553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 18:58:57.247971 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.247292 2553 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 18:58:57.252751 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252726 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 18:58:57.252751 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252746 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 18:58:57.252751 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252751 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 18:58:57.252751 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252754 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 18:58:57.252751 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252757 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 18:58:57.252751 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252760 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 18:58:57.252978 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252763 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 18:58:57.252978 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252766 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 18:58:57.252978 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252768 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 18:58:57.252978 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252771 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 18:58:57.252978 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252773 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 18:58:57.252978 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252776 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 18:58:57.252978 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252778 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 18:58:57.252978 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252781 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 18:58:57.252978 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252784 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 18:58:57.252978 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252787 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 18:58:57.252978 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252789 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 18:58:57.252978 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252792 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 18:58:57.252978 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252794 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 18:58:57.252978 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252799 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 18:58:57.252978 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252803 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 18:58:57.252978 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252805 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 18:58:57.252978 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252808 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 18:58:57.252978 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252811 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 18:58:57.252978 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252814 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 18:58:57.253444 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252818 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 18:58:57.253444 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252821 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 18:58:57.253444 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252823 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 18:58:57.253444 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252826 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 18:58:57.253444 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252829 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 18:58:57.253444 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252832 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 18:58:57.253444 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252834 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 18:58:57.253444 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252838 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 18:58:57.253444 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252840 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 18:58:57.253444 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252842 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 18:58:57.253444 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252845 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 18:58:57.253444 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252847 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 18:58:57.253444 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252850 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 18:58:57.253444 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252852 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 18:58:57.253444 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252856 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 18:58:57.253444 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252858 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 18:58:57.253444 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252861 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 18:58:57.253444 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252864 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 18:58:57.253444 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252867 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 18:58:57.253921 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252871 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 18:58:57.253921 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252875 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 06 18:58:57.253921 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252877 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 18:58:57.253921 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252880 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 18:58:57.253921 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252883 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 18:58:57.253921 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252885 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 18:58:57.253921 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252888 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 18:58:57.253921 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252890 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 18:58:57.253921 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252893 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 18:58:57.253921 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252896 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 18:58:57.253921 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252898 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 18:58:57.253921 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252901 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 18:58:57.253921 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252903 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 18:58:57.253921 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252906 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 18:58:57.253921 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252908 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 18:58:57.253921 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252911 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 18:58:57.253921 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252913 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 18:58:57.253921 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252916 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 18:58:57.253921 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252918 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252922 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252925 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252928 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252931 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252934 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252937 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252940 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252942 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252947 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252950 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252952 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252955 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252957 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252960 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252962 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252964 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252967 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252970 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252972 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 18:58:57.254377 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252975 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252977 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252980 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252983 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252985 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252988 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252990 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252993 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252996 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.252999 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253001 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253004 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253006 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253009 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253012 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253014 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253017 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253739 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253747 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253749 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 18:58:57.254879 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253752 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253755 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253758 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253761 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253763 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253766 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253769 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253772 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253774 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253777 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253780 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253782 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253785 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253787 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253791 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253793 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253796 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253798 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253801 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253803 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 18:58:57.255363 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253806 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 18:58:57.255871 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253808 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 18:58:57.255871 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253811 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 18:58:57.255871 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253813 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 18:58:57.255871 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253816 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 18:58:57.255871 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253818 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 18:58:57.255871 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253821 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 18:58:57.255871 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253823 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 18:58:57.255871 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253826 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 18:58:57.255871 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253828 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 18:58:57.255871 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253831 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 18:58:57.255871 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253834 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 18:58:57.255871 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253836 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 18:58:57.255871 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253840 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 18:58:57.255871 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253844 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 18:58:57.255871 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253848 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 18:58:57.255871 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253851 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 18:58:57.255871 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253853 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 18:58:57.255871 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253856 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 18:58:57.255871 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253858 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 18:58:57.256448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253861 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 18:58:57.256448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253863 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 18:58:57.256448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253866 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 18:58:57.256448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253868 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 18:58:57.256448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253871 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 18:58:57.256448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253873 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 18:58:57.256448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253876 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 18:58:57.256448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253878 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 18:58:57.256448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253881 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 18:58:57.256448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253884 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 18:58:57.256448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253886 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 18:58:57.256448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253889 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 18:58:57.256448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253891 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 18:58:57.256448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253894 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 18:58:57.256448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253896 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 18:58:57.256448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253899 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 18:58:57.256448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253901 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 18:58:57.256448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253904 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 18:58:57.256448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253906 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253909 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253912 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253914 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253918 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253922 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253925 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253928 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253932 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253935 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253937 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253940 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253942 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253945 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253947 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253950 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253953 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253955 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253958 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253960 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 18:58:57.256943 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253963 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253966 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253969 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253972 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253974 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253977 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253980 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253982 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253985 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253987 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253990 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253992 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253995 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.253997 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254000 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254002 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254006 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254009 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254376 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254382 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 18:58:57.257448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254385 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254387 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254390 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254394 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254397 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254400 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254404 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254406 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254409 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254412 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254414 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254417 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254419 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254422 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254425 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254428 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254430 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254433 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254435 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254438 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 18:58:57.257944 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254441 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 18:58:57.258442 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254444 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 18:58:57.258442 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254446 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 18:58:57.258442 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254448 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 18:58:57.258442 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254451 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 18:58:57.258442 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254453 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 18:58:57.258442 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254458 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 18:58:57.258442 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254462 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 18:58:57.258442 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254464 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 18:58:57.258442 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254467 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 18:58:57.258442 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254471 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 18:58:57.258442 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254474 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 18:58:57.258442 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254477 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 18:58:57.258442 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254479 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 18:58:57.258442 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254481 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 18:58:57.258442 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254484 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 18:58:57.258442 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254487 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 18:58:57.258442 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254489 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 18:58:57.258442 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254492 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 18:58:57.258442 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254494 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254497 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254499 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254502 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254504 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254507 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254509 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254512 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254514 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254517 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254519 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254521 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254524 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254526 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254529 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254531 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254536 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254538 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254540 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254543 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 18:58:57.258919 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254545 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 18:58:57.259414 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254548 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 06 18:58:57.259414 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254551 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 18:58:57.259414 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254554 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 18:58:57.259414 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254557 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 18:58:57.259414 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254560 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 18:58:57.259414 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254563 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 18:58:57.259414 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254566 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 18:58:57.259414 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254569 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 18:58:57.259414 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254572 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 18:58:57.259414 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254575 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 18:58:57.259414 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254578 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 18:58:57.259414 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254580 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 18:58:57.259414 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254583 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 18:58:57.259414 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254585 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 18:58:57.259414 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254588 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 18:58:57.259414 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254591 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 18:58:57.259414 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254594 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 18:58:57.259414 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254598 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 18:58:57.259414 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254601 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254604 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254607 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254609 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254626 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254629 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254632 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254634 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254637 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254641 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254644 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254646 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254649 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254651 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254654 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254656 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254659 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254662 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.254665 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254730 2553 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254737 2553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 18:58:57.259918 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254744 2553 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254748 2553 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254752 2553 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254754 2553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254759 2553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254762 2553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254766 2553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254769 2553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254772 2553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254775 2553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254778 2553 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254782 2553 flags.go:64] FLAG: --cgroup-root="" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254784 2553 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254787 2553 flags.go:64] FLAG: --client-ca-file="" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254791 2553 flags.go:64] FLAG: --cloud-provider="external" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254794 2553 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254797 2553 flags.go:64] FLAG: --cluster-domain="" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254800 2553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254803 2553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254806 2553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254811 2553 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254814 2553 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254817 2553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 18:58:57.260427 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254820 2553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254823 2553 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254826 2553 flags.go:64] FLAG: --contention-profiling="false" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254829 2553 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254831 2553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254834 2553 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254837 2553 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254841 2553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254844 2553 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254847 2553 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254849 2553 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254852 2553 flags.go:64] FLAG: --enable-server="true" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254855 2553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254859 2553 flags.go:64] FLAG: --event-burst="100" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254862 2553 flags.go:64] FLAG: --event-qps="50" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254864 2553 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254867 2553 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254870 2553 flags.go:64] FLAG: --eviction-hard="" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254873 2553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254876 2553 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254879 2553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254882 2553 flags.go:64] FLAG: --eviction-soft="" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254884 2553 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254887 2553 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254890 2553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 18:58:57.261035 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254893 2553 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254896 2553 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254899 2553 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254901 2553 flags.go:64] FLAG: --feature-gates="" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254908 2553 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254911 2553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254914 2553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254917 2553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254919 2553 flags.go:64] FLAG: --healthz-port="10248" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254922 2553 flags.go:64] FLAG: --help="false" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254925 2553 flags.go:64] FLAG: --hostname-override="ip-10-0-141-237.ec2.internal" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254928 2553 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254931 2553 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254934 2553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254937 2553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254940 2553 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254943 2553 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254945 2553 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254948 2553 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254950 2553 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254953 2553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254956 2553 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254959 2553 flags.go:64] FLAG: --kube-reserved="" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254962 2553 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 18:58:57.261670 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254965 2553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254967 2553 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254970 2553 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254973 2553 flags.go:64] FLAG: --lock-file="" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254975 2553 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254978 2553 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254981 2553 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254990 2553 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254993 2553 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254996 2553 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.254998 2553 flags.go:64] FLAG: --logging-format="text" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255001 2553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255005 2553 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255008 2553 flags.go:64] FLAG: --manifest-url="" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255010 2553 flags.go:64] FLAG: --manifest-url-header="" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255015 2553 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255017 2553 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255021 2553 flags.go:64] FLAG: --max-pods="110" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255024 2553 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255027 2553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255030 2553 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255032 2553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255035 2553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255038 2553 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255041 2553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 18:58:57.262298 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255048 2553 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255050 2553 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255053 2553 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255056 2553 flags.go:64] FLAG: --pod-cidr="" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255059 2553 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255064 2553 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255067 2553 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255070 2553 flags.go:64] FLAG: --pods-per-core="0" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255073 2553 flags.go:64] FLAG: --port="10250" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255076 2553 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255079 2553 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-067704a0f4244b423" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255082 2553 flags.go:64] FLAG: --qos-reserved="" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255084 2553 flags.go:64] FLAG: --read-only-port="10255" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255087 2553 flags.go:64] FLAG: --register-node="true" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255090 2553 flags.go:64] FLAG: --register-with-taints="" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255093 2553 flags.go:64] FLAG: --registry-burst="10" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255096 2553 flags.go:64] FLAG: --registry-qps="5" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255099 2553 flags.go:64] FLAG: --reserved-cpus="" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255102 2553 flags.go:64] FLAG: --reserved-memory="" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255105 2553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255110 2553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255113 2553 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255116 2553 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255119 2553 flags.go:64] FLAG: --runonce="false" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255121 2553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 18:58:57.262916 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255124 2553 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255127 2553 flags.go:64] FLAG: --seccomp-default="false" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255130 2553 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255132 2553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255136 2553 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255138 2553 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255141 2553 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255144 2553 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255147 2553 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255150 2553 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255152 2553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255155 2553 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255158 2553 flags.go:64] FLAG: --system-cgroups="" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255161 2553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255166 2553 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255168 2553 flags.go:64] FLAG: --tls-cert-file="" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255171 2553 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255174 2553 flags.go:64] FLAG: --tls-min-version="" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255177 2553 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255180 2553 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255182 2553 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255185 2553 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255188 2553 flags.go:64] FLAG: --v="2" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255192 2553 flags.go:64] FLAG: --version="false" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255196 2553 flags.go:64] FLAG: --vmodule="" Jul 06 18:58:57.263529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255200 2553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.255203 2553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255312 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255317 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255321 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255324 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255327 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255330 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255333 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255335 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255337 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255340 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255342 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255345 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255347 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255350 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255352 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255355 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255357 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255360 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 18:58:57.264145 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255363 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255366 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255368 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255371 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255374 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255376 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255379 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255381 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255383 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255386 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255389 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255391 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255397 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255401 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255404 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255407 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255410 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255412 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255414 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255417 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 18:58:57.264703 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255419 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 18:58:57.265184 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255423 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 18:58:57.265184 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255426 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 18:58:57.265184 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255429 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 18:58:57.265184 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255432 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 18:58:57.265184 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255435 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 18:58:57.265184 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255438 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 18:58:57.265184 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255441 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 18:58:57.265184 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255443 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 18:58:57.265184 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255446 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 18:58:57.265184 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255448 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 18:58:57.265184 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255451 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 18:58:57.265184 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255453 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 18:58:57.265184 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255455 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 18:58:57.265184 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255458 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 18:58:57.265184 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255461 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 18:58:57.265184 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255463 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 18:58:57.265184 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255466 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 18:58:57.265184 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255468 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 18:58:57.265184 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255471 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255473 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255476 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255479 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255483 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255486 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255490 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255493 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255496 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255499 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255501 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255504 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255506 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255509 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255511 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255514 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255516 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255519 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255521 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255523 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 18:58:57.265659 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255526 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255528 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255531 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255533 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255536 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255538 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255541 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255543 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255546 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255548 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255550 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255553 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255555 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255558 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255560 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255563 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255566 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255568 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255572 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255576 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 18:58:57.266157 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255579 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 18:58:57.266649 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.255581 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 18:58:57.266649 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.256275 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 18:58:57.266649 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.263578 2553 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 18:58:57.266649 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.263593 2553 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 18:58:57.266649 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.263606 2553 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 18:58:57.266649 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.263632 2553 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 18:58:57.266649 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263694 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 18:58:57.266649 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263699 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 18:58:57.266649 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263702 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 18:58:57.266649 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263705 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 18:58:57.266649 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263707 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 18:58:57.266649 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263710 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 18:58:57.266649 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263713 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 18:58:57.266649 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263716 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263719 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263721 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263724 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263726 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263729 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263733 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263737 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263741 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263744 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263747 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263750 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263752 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263755 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263758 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263760 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263763 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263765 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263768 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263770 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 18:58:57.267004 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263773 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263776 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263779 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263782 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263784 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263787 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263790 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263792 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263795 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263797 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263800 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263802 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263805 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263807 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263810 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263812 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263815 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263817 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263820 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263822 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 18:58:57.267487 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263825 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 18:58:57.267986 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263828 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 18:58:57.267986 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263830 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 18:58:57.267986 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263833 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 18:58:57.267986 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263835 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 18:58:57.267986 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263838 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 18:58:57.267986 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263840 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 18:58:57.267986 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263844 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 18:58:57.267986 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263848 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 18:58:57.267986 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263851 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 18:58:57.267986 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263854 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 18:58:57.267986 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263856 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 18:58:57.267986 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263859 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 18:58:57.267986 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263862 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 18:58:57.267986 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263864 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 18:58:57.267986 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263867 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 18:58:57.267986 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263870 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 18:58:57.267986 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263873 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 18:58:57.267986 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263875 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 18:58:57.267986 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263878 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263880 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263883 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263885 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263887 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263890 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263892 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263895 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263897 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263900 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263902 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263904 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263907 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263910 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263912 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263915 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263917 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263920 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263923 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263925 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 18:58:57.268448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263928 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 18:58:57.269065 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263930 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 18:58:57.269065 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263933 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 18:58:57.269065 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263937 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 18:58:57.269065 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263941 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 18:58:57.269065 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263944 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 18:58:57.269065 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263947 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 18:58:57.269065 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263949 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 18:58:57.269065 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263952 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 18:58:57.269065 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263955 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 18:58:57.269065 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263957 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 06 18:58:57.269065 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263960 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 18:58:57.269065 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.263962 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 18:58:57.269065 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.263968 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 18:58:57.269065 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265200 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 18:58:57.269065 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265207 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265210 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265214 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265217 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265219 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265222 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265226 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265230 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265233 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265236 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265239 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265242 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265244 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265247 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265249 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265252 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265254 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265257 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265259 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265263 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 18:58:57.269448 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265265 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265268 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265270 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265273 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265275 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265279 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265282 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265286 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265288 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265291 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265294 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265296 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265298 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265301 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265303 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265306 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265308 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265311 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265313 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265315 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 18:58:57.269959 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265318 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 18:58:57.270599 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265320 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 18:58:57.270599 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265323 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 18:58:57.270599 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265325 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 18:58:57.270599 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265328 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 18:58:57.270599 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265330 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 18:58:57.270599 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265333 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 18:58:57.270599 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265335 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 18:58:57.270599 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265338 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 18:58:57.270599 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265341 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 18:58:57.270599 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265343 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 18:58:57.270599 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265347 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 18:58:57.270599 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265351 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 18:58:57.270599 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265353 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 18:58:57.270599 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265356 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 18:58:57.270599 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265359 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 18:58:57.270599 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265362 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 18:58:57.270599 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265364 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 18:58:57.270599 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265367 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 18:58:57.270599 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265370 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265373 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265375 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265377 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265380 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265382 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265385 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265387 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265390 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265392 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265394 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265397 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265399 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265401 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265404 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265406 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265409 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265411 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265414 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265417 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 18:58:57.271084 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265419 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 18:58:57.271846 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265422 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 18:58:57.271846 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265424 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 18:58:57.271846 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265427 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 18:58:57.271846 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265429 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 18:58:57.271846 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265432 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 18:58:57.271846 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265434 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 18:58:57.271846 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265437 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 18:58:57.271846 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265439 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 18:58:57.271846 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265442 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 18:58:57.271846 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265444 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 18:58:57.271846 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265447 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 18:58:57.271846 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265449 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 18:58:57.271846 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265452 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 18:58:57.271846 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265455 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 18:58:57.271846 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265458 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 18:58:57.271846 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265460 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 18:58:57.271846 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265463 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 18:58:57.271846 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:57.265465 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 18:58:57.272479 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.265470 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 18:58:57.272479 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.265582 2553 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 18:58:57.272479 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.268533 2553 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 18:58:57.272479 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.269916 2553 server.go:1019] "Starting client certificate rotation" Jul 06 18:58:57.272479 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.270029 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 18:58:57.272479 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.270099 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 18:58:57.272479 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.272255 2553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 18:58:57.276387 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.276365 2553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 18:58:57.284865 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.284844 2553 log.go:25] "Validated CRI v1 runtime API" Jul 06 18:58:57.287111 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.287097 2553 log.go:25] "Validated CRI v1 image API" Jul 06 18:58:57.287910 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.287884 2553 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 18:58:57.293340 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.293322 2553 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 954aa18f-8e91-4921-a68a-857cce8d50fd:/dev/nvme0n1p4 f90bd9ae-9345-43e2-942f-e4e321c08369:/dev/nvme0n1p3] Jul 06 18:58:57.293390 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.293341 2553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 18:58:57.299163 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.299052 2553 manager.go:216] Machine: {Timestamp:2026-07-06 18:58:57.297446673 +0000 UTC m=+0.355429695 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097349 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d88501978e6eae13d178906e1846b SystemUUID:ec2d8850-1978-e6ea-e13d-178906e1846b BootID:b49d4ac9-c228-4675-bd63-9f60932f2797 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:53:8f:ce:87:8f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:53:8f:ce:87:8f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:68:f3:b2:13:c8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 18:58:57.299163 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.299156 2553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 18:58:57.299275 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.299228 2553 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 18:58:57.300458 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.300434 2553 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 18:58:57.300587 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.300461 2553 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-237.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 18:58:57.301399 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.301385 2553 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 18:58:57.301399 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.301399 2553 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 18:58:57.301479 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.301407 2553 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 18:58:57.302220 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.302209 2553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 18:58:57.303700 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.303676 2553 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 18:58:57.304483 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.304467 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 18:58:57.306728 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.306715 2553 state_mem.go:36] "Initialized new in-memory state store" Jul 06 18:58:57.306854 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.306845 2553 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 18:58:57.306894 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.306882 2553 kubelet.go:486] "Attempting to sync node with API server" Jul 06 18:58:57.306927 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.306896 2553 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 18:58:57.306927 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.306909 2553 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 18:58:57.306927 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.306918 2553 kubelet.go:398] "Adding apiserver pod source" Jul 06 18:58:57.306927 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.306927 2553 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 18:58:57.307280 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.307266 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 18:58:57.307334 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.307285 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 18:58:57.309706 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.309692 2553 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 18:58:57.312026 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.312013 2553 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 18:58:57.312121 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.312043 2553 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 18:58:57.313929 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.313915 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 18:58:57.313988 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.313935 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 18:58:57.313988 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.313941 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 18:58:57.313988 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.313947 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 18:58:57.313988 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.313959 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 18:58:57.313988 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.313976 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 18:58:57.313988 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.313986 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 18:58:57.314249 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.313992 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 18:58:57.314249 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.314002 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 18:58:57.314249 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.314011 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 18:58:57.314249 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.314024 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 18:58:57.314249 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.314033 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 18:58:57.314782 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.314768 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 18:58:57.314848 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.314802 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 18:58:57.319885 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.319870 2553 server.go:1268] "Started kubelet" Jul 06 18:58:57.320066 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.319987 2553 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 18:58:57.320066 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.320026 2553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 18:58:57.320229 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.320100 2553 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 18:58:57.320229 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.320194 2553 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-237.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 18:58:57.320351 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.320298 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-237.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 18:58:57.320351 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.320336 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 18:58:57.320679 ip-10-0-141-237 systemd[1]: Started Kubernetes Kubelet. Jul 06 18:58:57.322130 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.322113 2553 server.go:310] "Adding debug handlers to kubelet server" Jul 06 18:58:57.322246 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.322231 2553 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 18:58:57.324886 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.324868 2553 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 18:58:57.330468 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.330451 2553 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 18:58:57.330468 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.330461 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 18:58:57.330678 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.330640 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-237.ec2.internal\" not found" Jul 06 18:58:57.332025 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.331920 2553 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 18:58:57.332487 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.331919 2553 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 18:58:57.332487 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.332488 2553 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 18:58:57.332650 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.332179 2553 factory.go:55] Registering systemd factory Jul 06 18:58:57.332650 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.332554 2553 factory.go:223] Registration of the systemd container factory successfully Jul 06 18:58:57.332650 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.332605 2553 reconstruct.go:97] "Volume reconstruction finished" Jul 06 18:58:57.332650 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.332629 2553 reconciler.go:29] "Reconciler: start to sync state" Jul 06 18:58:57.333152 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.333108 2553 factory.go:153] Registering CRI-O factory Jul 06 18:58:57.333152 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.333124 2553 factory.go:223] Registration of the crio container factory successfully Jul 06 18:58:57.333285 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.333189 2553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 18:58:57.333285 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.333220 2553 factory.go:103] Registering Raw factory Jul 06 18:58:57.333285 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.333235 2553 manager.go:1196] Started watching for new ooms in manager Jul 06 18:58:57.333845 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.332409 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-237.ec2.internal.18bfc86a870c9aad default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-237.ec2.internal,UID:ip-10-0-141-237.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-237.ec2.internal,},FirstTimestamp:2026-07-06 18:58:57.319844525 +0000 UTC m=+0.377827546,LastTimestamp:2026-07-06 18:58:57.319844525 +0000 UTC m=+0.377827546,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-237.ec2.internal,}" Jul 06 18:58:57.333952 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.333843 2553 manager.go:349] Starting recovery of all containers Jul 06 18:58:57.344634 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.344475 2553 manager.go:354] Recovery completed Jul 06 18:58:57.345032 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.344998 2553 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-237.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 18:58:57.345163 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.345138 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 18:58:57.349542 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.349530 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 18:58:57.351260 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.351246 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasSufficientMemory" Jul 06 18:58:57.351304 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.351274 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 18:58:57.351304 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.351285 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasSufficientPID" Jul 06 18:58:57.351773 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.351760 2553 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 18:58:57.351773 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.351767 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 18:58:57.351864 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.351771 2553 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 18:58:57.351864 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.351840 2553 state_mem.go:36] "Initialized new in-memory state store" Jul 06 18:58:57.352392 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.352378 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasSufficientMemory" Jul 06 18:58:57.352462 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.352406 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 18:58:57.352462 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.352421 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasSufficientPID" Jul 06 18:58:57.352862 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.352850 2553 policy_none.go:49] "None policy: Start" Jul 06 18:58:57.352903 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.352871 2553 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 18:58:57.352903 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.352882 2553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 18:58:57.353770 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.353758 2553 policy_none.go:47] "Start" Jul 06 18:58:57.354670 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.354579 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-237.ec2.internal.18bfc86a88ebfc23 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-237.ec2.internal,UID:ip-10-0-141-237.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-237.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-237.ec2.internal,},FirstTimestamp:2026-07-06 18:58:57.351261219 +0000 UTC m=+0.409244240,LastTimestamp:2026-07-06 18:58:57.351261219 +0000 UTC m=+0.409244240,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-237.ec2.internal,}" Jul 06 18:58:57.367333 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.367268 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-237.ec2.internal.18bfc86a88ec4266 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-237.ec2.internal,UID:ip-10-0-141-237.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-237.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-237.ec2.internal,},FirstTimestamp:2026-07-06 18:58:57.351279206 +0000 UTC m=+0.409262226,LastTimestamp:2026-07-06 18:58:57.351279206 +0000 UTC m=+0.409262226,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-237.ec2.internal,}" Jul 06 18:58:57.382753 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.382686 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-237.ec2.internal.18bfc86a88ec684f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-237.ec2.internal,UID:ip-10-0-141-237.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-237.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-237.ec2.internal,},FirstTimestamp:2026-07-06 18:58:57.351288911 +0000 UTC m=+0.409271931,LastTimestamp:2026-07-06 18:58:57.351288911 +0000 UTC m=+0.409271931,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-237.ec2.internal,}" Jul 06 18:58:57.388312 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.388298 2553 manager.go:337] "Starting Device Plugin manager" Jul 06 18:58:57.400950 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.388342 2553 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 18:58:57.400950 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.388352 2553 server.go:87] "Starting device plugin registration server" Jul 06 18:58:57.400950 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.388597 2553 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 18:58:57.400950 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.388608 2553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 18:58:57.400950 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.388735 2553 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 18:58:57.400950 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.388822 2553 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 18:58:57.400950 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.388829 2553 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 18:58:57.400950 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.389299 2553 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 18:58:57.400950 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.389351 2553 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-237.ec2.internal\" not found" Jul 06 18:58:57.400950 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.392555 2553 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-237.ec2.internal.18bfc86a88ebfc23\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-237.ec2.internal.18bfc86a88ebfc23 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-237.ec2.internal,UID:ip-10-0-141-237.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-237.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-237.ec2.internal,},FirstTimestamp:2026-07-06 18:58:57.351261219 +0000 UTC m=+0.409244240,LastTimestamp:2026-07-06 18:58:57.35239168 +0000 UTC m=+0.410374704,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-237.ec2.internal,}" Jul 06 18:58:57.407914 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.407855 2553 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-237.ec2.internal.18bfc86a88ec4266\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-237.ec2.internal.18bfc86a88ec4266 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-237.ec2.internal,UID:ip-10-0-141-237.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-237.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-237.ec2.internal,},FirstTimestamp:2026-07-06 18:58:57.351279206 +0000 UTC m=+0.409262226,LastTimestamp:2026-07-06 18:58:57.35241306 +0000 UTC m=+0.410396082,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-237.ec2.internal,}" Jul 06 18:58:57.420808 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.420745 2553 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-237.ec2.internal.18bfc86a88ec684f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-237.ec2.internal.18bfc86a88ec684f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-237.ec2.internal,UID:ip-10-0-141-237.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-237.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-237.ec2.internal,},FirstTimestamp:2026-07-06 18:58:57.351288911 +0000 UTC m=+0.409271931,LastTimestamp:2026-07-06 18:58:57.352426125 +0000 UTC m=+0.410409148,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-237.ec2.internal,}" Jul 06 18:58:57.429600 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.429175 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-237.ec2.internal.18bfc86a8bc67fed default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-237.ec2.internal,UID:ip-10-0-141-237.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-141-237.ec2.internal,},FirstTimestamp:2026-07-06 18:58:57.399136237 +0000 UTC m=+0.457119250,LastTimestamp:2026-07-06 18:58:57.399136237 +0000 UTC m=+0.457119250,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-237.ec2.internal,}" Jul 06 18:58:57.445777 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.445757 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-snjn6" Jul 06 18:58:57.456229 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.456215 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-snjn6" Jul 06 18:58:57.486152 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.486129 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 18:58:57.487375 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.487356 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 18:58:57.487421 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.487403 2553 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 18:58:57.487452 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.487425 2553 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 18:58:57.487491 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.487463 2553 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 18:58:57.488867 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.488851 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 18:58:57.489698 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.489683 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasSufficientMemory" Jul 06 18:58:57.489779 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.489712 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 18:58:57.489779 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.489722 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasSufficientPID" Jul 06 18:58:57.489779 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.489747 2553 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-237.ec2.internal" Jul 06 18:58:57.501241 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.501188 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 18:58:57.521559 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.521536 2553 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-237.ec2.internal" Jul 06 18:58:57.521559 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.521556 2553 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-237.ec2.internal\": node \"ip-10-0-141-237.ec2.internal\" not found" Jul 06 18:58:57.588470 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.588442 2553 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-237.ec2.internal"] Jul 06 18:58:57.588562 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.588527 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 18:58:57.589770 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.589757 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasSufficientMemory" Jul 06 18:58:57.589859 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.589786 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 18:58:57.589859 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.589800 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasSufficientPID" Jul 06 18:58:57.590886 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.590872 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 18:58:57.590999 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.590977 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal" Jul 06 18:58:57.591046 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.591007 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 18:58:57.591501 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.591478 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasSufficientMemory" Jul 06 18:58:57.591573 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.591504 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 18:58:57.591573 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.591517 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasSufficientPID" Jul 06 18:58:57.591573 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.591541 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasSufficientMemory" Jul 06 18:58:57.591573 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.591560 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 18:58:57.591573 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.591573 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasSufficientPID" Jul 06 18:58:57.592627 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.592598 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-237.ec2.internal" Jul 06 18:58:57.592683 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.592640 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 18:58:57.593177 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.593161 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasSufficientMemory" Jul 06 18:58:57.593271 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.593190 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 18:58:57.593271 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.593215 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeHasSufficientPID" Jul 06 18:58:57.593365 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.593349 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-237.ec2.internal\" not found" Jul 06 18:58:57.617165 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.617143 2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-237.ec2.internal\" not found" node="ip-10-0-141-237.ec2.internal" Jul 06 18:58:57.621482 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.621469 2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-237.ec2.internal\" not found" node="ip-10-0-141-237.ec2.internal" Jul 06 18:58:57.634524 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.634504 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6f5c007a1cc8c7f01af448c594df5b2e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal\" (UID: \"6f5c007a1cc8c7f01af448c594df5b2e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal" Jul 06 18:58:57.634573 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.634529 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f5c007a1cc8c7f01af448c594df5b2e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal\" (UID: \"6f5c007a1cc8c7f01af448c594df5b2e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal" Jul 06 18:58:57.634573 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.634548 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/62e528d50850ba268ed5e671edfd2eb2-config\") pod \"kube-apiserver-proxy-ip-10-0-141-237.ec2.internal\" (UID: \"62e528d50850ba268ed5e671edfd2eb2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-237.ec2.internal" Jul 06 18:58:57.694362 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.694337 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-237.ec2.internal\" not found" Jul 06 18:58:57.734675 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.734651 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6f5c007a1cc8c7f01af448c594df5b2e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal\" (UID: \"6f5c007a1cc8c7f01af448c594df5b2e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal" Jul 06 18:58:57.734742 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.734683 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6f5c007a1cc8c7f01af448c594df5b2e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal\" (UID: \"6f5c007a1cc8c7f01af448c594df5b2e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal" Jul 06 18:58:57.734742 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.734691 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f5c007a1cc8c7f01af448c594df5b2e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal\" (UID: \"6f5c007a1cc8c7f01af448c594df5b2e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal" Jul 06 18:58:57.734742 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.734715 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/62e528d50850ba268ed5e671edfd2eb2-config\") pod \"kube-apiserver-proxy-ip-10-0-141-237.ec2.internal\" (UID: \"62e528d50850ba268ed5e671edfd2eb2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-237.ec2.internal" Jul 06 18:58:57.734742 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.734736 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f5c007a1cc8c7f01af448c594df5b2e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal\" (UID: \"6f5c007a1cc8c7f01af448c594df5b2e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal" Jul 06 18:58:57.734871 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.734743 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/62e528d50850ba268ed5e671edfd2eb2-config\") pod \"kube-apiserver-proxy-ip-10-0-141-237.ec2.internal\" (UID: \"62e528d50850ba268ed5e671edfd2eb2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-237.ec2.internal" Jul 06 18:58:57.794692 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.794644 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-237.ec2.internal\" not found" Jul 06 18:58:57.894730 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.894708 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-237.ec2.internal\" not found" Jul 06 18:58:57.919873 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.919855 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal" Jul 06 18:58:57.924482 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:57.924466 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-237.ec2.internal" Jul 06 18:58:57.995299 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:57.995272 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-237.ec2.internal\" not found" Jul 06 18:58:58.095875 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:58.095811 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-237.ec2.internal\" not found" Jul 06 18:58:58.196427 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:58.196402 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-237.ec2.internal\" not found" Jul 06 18:58:58.269000 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.268965 2553 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 18:58:58.269548 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.269144 2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 18:58:58.296478 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:58.296459 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-237.ec2.internal\" not found" Jul 06 18:58:58.296636 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.296602 2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 18:58:58.307802 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.307784 2553 apiserver.go:52] "Watching apiserver" Jul 06 18:58:58.317700 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.317678 2553 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 18:58:58.318004 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.317983 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-jmvjq","openshift-network-diagnostics/network-check-target-hklfk","openshift-network-operator/iptables-alerter-842q9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g","openshift-cluster-node-tuning-operator/tuned-fl6nt","openshift-image-registry/node-ca-v2xjc","openshift-ovn-kubernetes/ovnkube-node-x5g5z","kube-system/global-pull-secret-syncer-v5xs7","kube-system/konnectivity-agent-6gm25","openshift-dns/node-resolver-nb9d8","openshift-multus/multus-additional-cni-plugins-vlq9c","openshift-multus/multus-vzw9m"] Jul 06 18:58:58.319510 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.319498 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:58:58.319571 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:58.319555 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmvjq" podUID="e7b6d1cb-2542-4861-9475-25365c8a5d15" Jul 06 18:58:58.320679 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.320666 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:58:58.320724 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:58.320712 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hklfk" podUID="aab0c2d2-909e-466d-bd99-c4f4b3830060" Jul 06 18:58:58.321786 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.321770 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-842q9" Jul 06 18:58:58.322941 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.322920 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.324440 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.324424 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.325705 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.325688 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v2xjc" Jul 06 18:58:58.327098 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.327079 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.328528 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.328512 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:58:58.328593 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:58.328562 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5xs7" podUID="f7d2aeda-aa6f-4865-ade0-f9aa24ea785a" Jul 06 18:58:58.329895 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.329881 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6gm25" Jul 06 18:58:58.330608 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.330564 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 18:58:58.330608 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.330586 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-spxrn\"" Jul 06 18:58:58.330608 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.330568 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 18:58:58.330773 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.330591 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal" Jul 06 18:58:58.331144 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.331131 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nb9d8" Jul 06 18:58:58.331667 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.331649 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 18:58:58.331667 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.331658 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7kq2n\"" Jul 06 18:58:58.331805 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.331651 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 18:58:58.331805 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.331784 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 18:58:58.331879 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.331808 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 18:58:58.331922 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.331909 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 18:58:58.332597 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.332583 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.333782 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.333765 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.335167 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.335150 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 18:58:58.335238 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.335195 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kc5w5\"" Jul 06 18:58:58.335981 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.335969 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 18:58:58.337102 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337086 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-host\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.337176 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337114 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dd572f4-3c97-402e-8480-6209afbdf600-tmp\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.337176 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337138 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-run-netns\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.337176 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337162 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-system-cni-dir\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.337547 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337184 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/551e1a05-81d4-46f4-9109-3654599f47c1-etc-selinux\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.337547 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337439 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27f875a5-8c79-42e8-b05f-819fe5dcc452-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.337547 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337466 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27f875a5-8c79-42e8-b05f-819fe5dcc452-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.337547 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337489 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-run-systemd\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.337547 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337511 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-run-ovn\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.337547 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337534 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-cni-bin\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.337818 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337554 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d422e92a-be96-4d4a-8011-66ff2945286d-env-overrides\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.337818 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337594 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-run-ovn-kubernetes\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.337818 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337643 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/551e1a05-81d4-46f4-9109-3654599f47c1-device-dir\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.337818 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337660 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-dbus\") pod \"global-pull-secret-syncer-v5xs7\" (UID: \"f7d2aeda-aa6f-4865-ade0-f9aa24ea785a\") " pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:58:58.337818 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337674 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99098e94-f6ae-4918-b007-e85999dfe259-konnectivity-ca\") pod \"konnectivity-agent-6gm25\" (UID: \"99098e94-f6ae-4918-b007-e85999dfe259\") " pod="kube-system/konnectivity-agent-6gm25" Jul 06 18:58:58.337818 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337689 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/551e1a05-81d4-46f4-9109-3654599f47c1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.337818 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337704 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-etc-systemd\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.337818 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337719 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d422e92a-be96-4d4a-8011-66ff2945286d-ovn-node-metrics-cert\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.337818 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337733 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5nsw\" (UniqueName: \"kubernetes.io/projected/d422e92a-be96-4d4a-8011-66ff2945286d-kube-api-access-b5nsw\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.337818 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337747 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99098e94-f6ae-4918-b007-e85999dfe259-agent-certs\") pod \"konnectivity-agent-6gm25\" (UID: \"99098e94-f6ae-4918-b007-e85999dfe259\") " pod="kube-system/konnectivity-agent-6gm25" Jul 06 18:58:58.337818 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337761 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27f875a5-8c79-42e8-b05f-819fe5dcc452-system-cni-dir\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.337818 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337774 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-slash\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.337818 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337788 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-cni-netd\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.337818 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337803 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-host-run-netns\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.337818 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337817 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-host-var-lib-cni-bin\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.338527 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337857 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-multus-conf-dir\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.338527 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337889 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c996c2ca-cb6f-420e-b4d0-5b9720137b7d-serviceca\") pod \"node-ca-v2xjc\" (UID: \"c996c2ca-cb6f-420e-b4d0-5b9720137b7d\") " pod="openshift-image-registry/node-ca-v2xjc" Jul 06 18:58:58.338527 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337920 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/27f875a5-8c79-42e8-b05f-819fe5dcc452-os-release\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.338527 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337959 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7kkf\" (UniqueName: \"kubernetes.io/projected/8dd572f4-3c97-402e-8480-6209afbdf600-kube-api-access-s7kkf\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.338527 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.337984 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-multus-socket-dir-parent\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.338527 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338022 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-host-var-lib-cni-multus\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.338527 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338038 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmgrx\" (UniqueName: \"kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx\") pod \"network-check-target-hklfk\" (UID: \"aab0c2d2-909e-466d-bd99-c4f4b3830060\") " pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:58:58.338527 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338066 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c996c2ca-cb6f-420e-b4d0-5b9720137b7d-host\") pod \"node-ca-v2xjc\" (UID: \"c996c2ca-cb6f-420e-b4d0-5b9720137b7d\") " pod="openshift-image-registry/node-ca-v2xjc" Jul 06 18:58:58.338527 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338089 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/551e1a05-81d4-46f4-9109-3654599f47c1-registration-dir\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.338527 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338113 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-lib-modules\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.338527 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338138 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-log-socket\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.338527 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338164 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-multus-cni-dir\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.338527 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338179 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-etc-sysctl-conf\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.338527 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338193 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-var-lib-openvswitch\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.338527 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338215 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.338527 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338239 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-kubelet-config\") pod \"global-pull-secret-syncer-v5xs7\" (UID: \"f7d2aeda-aa6f-4865-ade0-f9aa24ea785a\") " pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:58:58.339407 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338256 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27f875a5-8c79-42e8-b05f-819fe5dcc452-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.339407 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338287 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggl6v\" (UniqueName: \"kubernetes.io/projected/e7b6d1cb-2542-4861-9475-25365c8a5d15-kube-api-access-ggl6v\") pod \"network-metrics-daemon-jmvjq\" (UID: \"e7b6d1cb-2542-4861-9475-25365c8a5d15\") " pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:58:58.339407 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338321 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-systemd-units\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.339407 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338351 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f70581d-708c-43ae-871a-a126644cefff-hosts-file\") pod \"node-resolver-nb9d8\" (UID: \"2f70581d-708c-43ae-871a-a126644cefff\") " pod="openshift-dns/node-resolver-nb9d8" Jul 06 18:58:58.339407 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338377 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-etc-kubernetes\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.339407 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338418 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb86a840-dfb9-4800-9a40-080862ec9018-host-slash\") pod \"iptables-alerter-842q9\" (UID: \"eb86a840-dfb9-4800-9a40-080862ec9018\") " pod="openshift-network-operator/iptables-alerter-842q9" Jul 06 18:58:58.339407 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338456 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d422e92a-be96-4d4a-8011-66ff2945286d-ovnkube-script-lib\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.339407 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338484 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-cnibin\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.339407 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338503 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27f875a5-8c79-42e8-b05f-819fe5dcc452-cnibin\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.339407 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338523 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27f875a5-8c79-42e8-b05f-819fe5dcc452-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.339407 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338540 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5gck\" (UniqueName: \"kubernetes.io/projected/27f875a5-8c79-42e8-b05f-819fe5dcc452-kube-api-access-w5gck\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.339407 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338554 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb86a840-dfb9-4800-9a40-080862ec9018-iptables-alerter-script\") pod \"iptables-alerter-842q9\" (UID: \"eb86a840-dfb9-4800-9a40-080862ec9018\") " pod="openshift-network-operator/iptables-alerter-842q9" Jul 06 18:58:58.339407 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338569 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d945k\" (UniqueName: \"kubernetes.io/projected/eb86a840-dfb9-4800-9a40-080862ec9018-kube-api-access-d945k\") pod \"iptables-alerter-842q9\" (UID: \"eb86a840-dfb9-4800-9a40-080862ec9018\") " pod="openshift-network-operator/iptables-alerter-842q9" Jul 06 18:58:58.339407 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338587 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-etc-kubernetes\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.339407 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338602 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d422e92a-be96-4d4a-8011-66ff2945286d-ovnkube-config\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.339407 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338643 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs\") pod \"network-metrics-daemon-jmvjq\" (UID: \"e7b6d1cb-2542-4861-9475-25365c8a5d15\") " pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:58:58.340180 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338683 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-etc-sysconfig\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.340180 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338715 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-var-lib-kubelet\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.340180 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338740 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-etc-openvswitch\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.340180 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338782 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-etc-modprobe-d\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.340180 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338795 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-run\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.340180 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338811 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-kubelet\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.340180 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338838 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpkjh\" (UniqueName: \"kubernetes.io/projected/2f70581d-708c-43ae-871a-a126644cefff-kube-api-access-cpkjh\") pod \"node-resolver-nb9d8\" (UID: \"2f70581d-708c-43ae-871a-a126644cefff\") " pod="openshift-dns/node-resolver-nb9d8" Jul 06 18:58:58.340180 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338855 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0dbfe656-ce52-4013-9445-7adb45ee8fb8-multus-daemon-config\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.340180 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338868 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dbfe656-ce52-4013-9445-7adb45ee8fb8-cni-binary-copy\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.340180 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338885 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-host-run-k8s-cni-cncf-io\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.340180 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338868 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 18:58:58.340180 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338907 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-host-run-multus-certs\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.340180 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338932 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8n7g\" (UniqueName: \"kubernetes.io/projected/551e1a05-81d4-46f4-9109-3654599f47c1-kube-api-access-w8n7g\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.340180 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.338948 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-run-openvswitch\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.340180 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.339003 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-node-log\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.340180 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.339025 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqrlf\" (UniqueName: \"kubernetes.io/projected/c996c2ca-cb6f-420e-b4d0-5b9720137b7d-kube-api-access-pqrlf\") pod \"node-ca-v2xjc\" (UID: \"c996c2ca-cb6f-420e-b4d0-5b9720137b7d\") " pod="openshift-image-registry/node-ca-v2xjc" Jul 06 18:58:58.340180 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.339052 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret\") pod \"global-pull-secret-syncer-v5xs7\" (UID: \"f7d2aeda-aa6f-4865-ade0-f9aa24ea785a\") " pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:58:58.340989 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.339067 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/551e1a05-81d4-46f4-9109-3654599f47c1-socket-dir\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.340989 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.339081 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-host-var-lib-kubelet\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.340989 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.339094 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-hostroot\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.340989 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.339107 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqz8t\" (UniqueName: \"kubernetes.io/projected/0dbfe656-ce52-4013-9445-7adb45ee8fb8-kube-api-access-sqz8t\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.340989 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.339108 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 18:58:58.340989 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.339121 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/551e1a05-81d4-46f4-9109-3654599f47c1-sys-fs\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.340989 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.339156 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-etc-sysctl-d\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.340989 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.339171 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8dd572f4-3c97-402e-8480-6209afbdf600-etc-tuned\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.340989 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.339184 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f70581d-708c-43ae-871a-a126644cefff-tmp-dir\") pod \"node-resolver-nb9d8\" (UID: \"2f70581d-708c-43ae-871a-a126644cefff\") " pod="openshift-dns/node-resolver-nb9d8" Jul 06 18:58:58.340989 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.339205 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-os-release\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.340989 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.339231 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-sys\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.343242 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.343220 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 18:58:58.351919 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.351860 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wz5kg\"" Jul 06 18:58:58.351919 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.351867 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 18:58:58.351919 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.351894 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 18:58:58.352092 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.351942 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 18:58:58.352092 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.351880 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 18:58:58.352838 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.352815 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 18:58:58.352919 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.352857 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 18:58:58.353005 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.352998 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 18:58:58.353125 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.353107 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 18:58:58.353204 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.353178 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 18:58:58.353291 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.353275 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 18:58:58.353469 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.353451 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 18:58:58.353549 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.353474 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 18:58:58.353549 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.353510 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 18:58:58.353672 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.353661 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wxkcb\"" Jul 06 18:58:58.353975 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.353909 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 18:58:58.354199 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.354184 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9wfqv\"" Jul 06 18:58:58.354435 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.354342 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 18:58:58.354598 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.354583 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h4dbz\"" Jul 06 18:58:58.354733 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.354600 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qxkdf\"" Jul 06 18:58:58.354733 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.354663 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 18:58:58.354866 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.354846 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-czhzt\"" Jul 06 18:58:58.377048 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.377032 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal"] Jul 06 18:58:58.377892 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.377876 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 18:58:58.377987 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.377946 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-237.ec2.internal" Jul 06 18:58:58.382887 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.382860 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 18:58:58.424152 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.424132 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 18:58:58.424328 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.424313 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-237.ec2.internal"] Jul 06 18:58:58.425520 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.425489 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-f28sx" Jul 06 18:58:58.433029 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.433015 2553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 18:58:58.439887 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.439867 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/551e1a05-81d4-46f4-9109-3654599f47c1-registration-dir\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.439986 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.439902 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-lib-modules\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.439986 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.439930 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-log-socket\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.439986 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.439953 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-multus-cni-dir\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.439986 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.439975 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-etc-sysctl-conf\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.440192 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.439995 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-var-lib-openvswitch\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.440192 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.439997 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/551e1a05-81d4-46f4-9109-3654599f47c1-registration-dir\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.440192 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440010 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-log-socket\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.440192 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440019 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.440192 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440037 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-multus-cni-dir\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.440192 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440049 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-var-lib-openvswitch\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.440192 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440051 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-lib-modules\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.440192 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440062 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.440192 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440060 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-kubelet-config\") pod \"global-pull-secret-syncer-v5xs7\" (UID: \"f7d2aeda-aa6f-4865-ade0-f9aa24ea785a\") " pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:58:58.440192 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440101 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-kubelet-config\") pod \"global-pull-secret-syncer-v5xs7\" (UID: \"f7d2aeda-aa6f-4865-ade0-f9aa24ea785a\") " pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:58:58.440192 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440107 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27f875a5-8c79-42e8-b05f-819fe5dcc452-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.440192 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440114 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-etc-sysctl-conf\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.440192 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440139 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggl6v\" (UniqueName: \"kubernetes.io/projected/e7b6d1cb-2542-4861-9475-25365c8a5d15-kube-api-access-ggl6v\") pod \"network-metrics-daemon-jmvjq\" (UID: \"e7b6d1cb-2542-4861-9475-25365c8a5d15\") " pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:58:58.440192 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440164 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-systemd-units\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.440192 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440188 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f70581d-708c-43ae-871a-a126644cefff-hosts-file\") pod \"node-resolver-nb9d8\" (UID: \"2f70581d-708c-43ae-871a-a126644cefff\") " pod="openshift-dns/node-resolver-nb9d8" Jul 06 18:58:58.440959 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440207 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-systemd-units\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.440959 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440210 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-etc-kubernetes\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.440959 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440256 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-etc-kubernetes\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.440959 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440281 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f70581d-708c-43ae-871a-a126644cefff-hosts-file\") pod \"node-resolver-nb9d8\" (UID: \"2f70581d-708c-43ae-871a-a126644cefff\") " pod="openshift-dns/node-resolver-nb9d8" Jul 06 18:58:58.440959 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440284 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb86a840-dfb9-4800-9a40-080862ec9018-host-slash\") pod \"iptables-alerter-842q9\" (UID: \"eb86a840-dfb9-4800-9a40-080862ec9018\") " pod="openshift-network-operator/iptables-alerter-842q9" Jul 06 18:58:58.440959 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440308 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d422e92a-be96-4d4a-8011-66ff2945286d-ovnkube-script-lib\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.440959 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440333 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-cnibin\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.440959 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440357 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27f875a5-8c79-42e8-b05f-819fe5dcc452-cnibin\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.440959 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440372 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb86a840-dfb9-4800-9a40-080862ec9018-host-slash\") pod \"iptables-alerter-842q9\" (UID: \"eb86a840-dfb9-4800-9a40-080862ec9018\") " pod="openshift-network-operator/iptables-alerter-842q9" Jul 06 18:58:58.440959 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440382 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27f875a5-8c79-42e8-b05f-819fe5dcc452-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.440959 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440408 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5gck\" (UniqueName: \"kubernetes.io/projected/27f875a5-8c79-42e8-b05f-819fe5dcc452-kube-api-access-w5gck\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.440959 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440412 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-cnibin\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.440959 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440432 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27f875a5-8c79-42e8-b05f-819fe5dcc452-cnibin\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.440959 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440434 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb86a840-dfb9-4800-9a40-080862ec9018-iptables-alerter-script\") pod \"iptables-alerter-842q9\" (UID: \"eb86a840-dfb9-4800-9a40-080862ec9018\") " pod="openshift-network-operator/iptables-alerter-842q9" Jul 06 18:58:58.440959 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440484 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d945k\" (UniqueName: \"kubernetes.io/projected/eb86a840-dfb9-4800-9a40-080862ec9018-kube-api-access-d945k\") pod \"iptables-alerter-842q9\" (UID: \"eb86a840-dfb9-4800-9a40-080862ec9018\") " pod="openshift-network-operator/iptables-alerter-842q9" Jul 06 18:58:58.440959 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440508 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-etc-kubernetes\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.440959 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440532 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d422e92a-be96-4d4a-8011-66ff2945286d-ovnkube-config\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.441583 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440558 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs\") pod \"network-metrics-daemon-jmvjq\" (UID: \"e7b6d1cb-2542-4861-9475-25365c8a5d15\") " pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:58:58.441583 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440579 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-etc-sysconfig\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.441583 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440634 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-var-lib-kubelet\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.441583 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440699 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-var-lib-kubelet\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.441583 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:58.440785 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 18:58:58.441583 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440790 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27f875a5-8c79-42e8-b05f-819fe5dcc452-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.441583 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440825 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-etc-kubernetes\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.441583 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440785 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-etc-sysconfig\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.441583 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:58.440854 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs podName:e7b6d1cb-2542-4861-9475-25365c8a5d15 nodeName:}" failed. No retries permitted until 2026-07-06 18:58:58.940834823 +0000 UTC m=+1.998817848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs") pod "network-metrics-daemon-jmvjq" (UID: "e7b6d1cb-2542-4861-9475-25365c8a5d15") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 18:58:58.441583 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440875 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-etc-openvswitch\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.441583 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440895 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-etc-openvswitch\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.441583 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440903 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-etc-modprobe-d\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.441583 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440929 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-run\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.441583 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440952 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-kubelet\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.441583 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440969 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27f875a5-8c79-42e8-b05f-819fe5dcc452-cni-binary-copy\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.441583 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.440975 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpkjh\" (UniqueName: \"kubernetes.io/projected/2f70581d-708c-43ae-871a-a126644cefff-kube-api-access-cpkjh\") pod \"node-resolver-nb9d8\" (UID: \"2f70581d-708c-43ae-871a-a126644cefff\") " pod="openshift-dns/node-resolver-nb9d8" Jul 06 18:58:58.441583 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441011 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eb86a840-dfb9-4800-9a40-080862ec9018-iptables-alerter-script\") pod \"iptables-alerter-842q9\" (UID: \"eb86a840-dfb9-4800-9a40-080862ec9018\") " pod="openshift-network-operator/iptables-alerter-842q9" Jul 06 18:58:58.442408 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441018 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-kubelet\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.442408 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441056 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d422e92a-be96-4d4a-8011-66ff2945286d-ovnkube-script-lib\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.442408 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441069 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-etc-modprobe-d\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.442408 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441079 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-run\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.442408 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441059 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0dbfe656-ce52-4013-9445-7adb45ee8fb8-multus-daemon-config\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.442408 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441111 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dbfe656-ce52-4013-9445-7adb45ee8fb8-cni-binary-copy\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.442408 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441135 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-host-run-k8s-cni-cncf-io\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.442408 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441157 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-host-run-multus-certs\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.442408 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441179 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8n7g\" (UniqueName: \"kubernetes.io/projected/551e1a05-81d4-46f4-9109-3654599f47c1-kube-api-access-w8n7g\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.442408 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441205 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-host-run-k8s-cni-cncf-io\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.442408 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441231 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-host-run-multus-certs\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.442408 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441244 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-run-openvswitch\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.442408 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441213 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-run-openvswitch\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.442408 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441273 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-node-log\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.442408 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441311 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqrlf\" (UniqueName: \"kubernetes.io/projected/c996c2ca-cb6f-420e-b4d0-5b9720137b7d-kube-api-access-pqrlf\") pod \"node-ca-v2xjc\" (UID: \"c996c2ca-cb6f-420e-b4d0-5b9720137b7d\") " pod="openshift-image-registry/node-ca-v2xjc" Jul 06 18:58:58.442408 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441354 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret\") pod \"global-pull-secret-syncer-v5xs7\" (UID: \"f7d2aeda-aa6f-4865-ade0-f9aa24ea785a\") " pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:58:58.442408 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441361 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-node-log\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.443216 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441386 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/551e1a05-81d4-46f4-9109-3654599f47c1-socket-dir\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.443216 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441412 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-host-var-lib-kubelet\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.443216 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:58.441428 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 18:58:58.443216 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441435 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-hostroot\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.443216 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:58.441484 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret podName:f7d2aeda-aa6f-4865-ade0-f9aa24ea785a nodeName:}" failed. No retries permitted until 2026-07-06 18:58:58.941465828 +0000 UTC m=+1.999448852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret") pod "global-pull-secret-syncer-v5xs7" (UID: "f7d2aeda-aa6f-4865-ade0-f9aa24ea785a") : object "kube-system"/"original-pull-secret" not registered Jul 06 18:58:58.443216 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441483 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqz8t\" (UniqueName: \"kubernetes.io/projected/0dbfe656-ce52-4013-9445-7adb45ee8fb8-kube-api-access-sqz8t\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.443216 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441510 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d422e92a-be96-4d4a-8011-66ff2945286d-ovnkube-config\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.443216 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441514 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/551e1a05-81d4-46f4-9109-3654599f47c1-sys-fs\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.443216 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441525 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/551e1a05-81d4-46f4-9109-3654599f47c1-socket-dir\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.443216 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441556 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-etc-sysctl-d\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.443216 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441559 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/551e1a05-81d4-46f4-9109-3654599f47c1-sys-fs\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.443216 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441579 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8dd572f4-3c97-402e-8480-6209afbdf600-etc-tuned\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.443216 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441601 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f70581d-708c-43ae-871a-a126644cefff-tmp-dir\") pod \"node-resolver-nb9d8\" (UID: \"2f70581d-708c-43ae-871a-a126644cefff\") " pod="openshift-dns/node-resolver-nb9d8" Jul 06 18:58:58.443216 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441558 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-hostroot\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.443216 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441640 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-os-release\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.443216 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441661 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-sys\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.443216 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441693 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-host\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.444051 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441703 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-host-var-lib-kubelet\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.444051 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441715 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dd572f4-3c97-402e-8480-6209afbdf600-tmp\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.444051 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441713 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-etc-sysctl-d\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.444051 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441742 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-run-netns\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.444051 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441780 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-run-netns\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.444051 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441799 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-sys\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.444051 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441828 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-host\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.444051 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441801 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-os-release\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.444051 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441857 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-system-cni-dir\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.444051 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441885 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/551e1a05-81d4-46f4-9109-3654599f47c1-etc-selinux\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.444051 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441891 2553 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 18:58:58.444051 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441909 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f70581d-708c-43ae-871a-a126644cefff-tmp-dir\") pod \"node-resolver-nb9d8\" (UID: \"2f70581d-708c-43ae-871a-a126644cefff\") " pod="openshift-dns/node-resolver-nb9d8" Jul 06 18:58:58.444051 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441912 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27f875a5-8c79-42e8-b05f-819fe5dcc452-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.444051 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441972 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-system-cni-dir\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.444051 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.441979 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27f875a5-8c79-42e8-b05f-819fe5dcc452-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.444051 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442014 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-run-systemd\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.444051 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442033 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27f875a5-8c79-42e8-b05f-819fe5dcc452-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.444051 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442041 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-run-ovn\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.445040 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442052 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/551e1a05-81d4-46f4-9109-3654599f47c1-etc-selinux\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.445040 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442066 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-cni-bin\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.445040 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442090 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d422e92a-be96-4d4a-8011-66ff2945286d-env-overrides\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.445040 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442096 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-run-systemd\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.445040 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442114 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-run-ovn-kubernetes\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.445040 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442126 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0dbfe656-ce52-4013-9445-7adb45ee8fb8-multus-daemon-config\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.445040 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442139 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/551e1a05-81d4-46f4-9109-3654599f47c1-device-dir\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.445040 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442140 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-cni-bin\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.445040 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442173 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-run-ovn\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.445040 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442176 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-dbus\") pod \"global-pull-secret-syncer-v5xs7\" (UID: \"f7d2aeda-aa6f-4865-ade0-f9aa24ea785a\") " pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:58:58.445040 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442208 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/551e1a05-81d4-46f4-9109-3654599f47c1-device-dir\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.445040 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442211 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99098e94-f6ae-4918-b007-e85999dfe259-konnectivity-ca\") pod \"konnectivity-agent-6gm25\" (UID: \"99098e94-f6ae-4918-b007-e85999dfe259\") " pod="kube-system/konnectivity-agent-6gm25" Jul 06 18:58:58.445040 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442260 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/551e1a05-81d4-46f4-9109-3654599f47c1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.445040 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442304 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-etc-systemd\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.445040 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442330 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d422e92a-be96-4d4a-8011-66ff2945286d-ovn-node-metrics-cert\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.445040 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442343 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-dbus\") pod \"global-pull-secret-syncer-v5xs7\" (UID: \"f7d2aeda-aa6f-4865-ade0-f9aa24ea785a\") " pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:58:58.445040 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442352 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5nsw\" (UniqueName: \"kubernetes.io/projected/d422e92a-be96-4d4a-8011-66ff2945286d-kube-api-access-b5nsw\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.445695 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442378 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99098e94-f6ae-4918-b007-e85999dfe259-agent-certs\") pod \"konnectivity-agent-6gm25\" (UID: \"99098e94-f6ae-4918-b007-e85999dfe259\") " pod="kube-system/konnectivity-agent-6gm25" Jul 06 18:58:58.445695 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442388 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8dd572f4-3c97-402e-8480-6209afbdf600-etc-systemd\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.445695 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442401 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27f875a5-8c79-42e8-b05f-819fe5dcc452-system-cni-dir\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.445695 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442424 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/551e1a05-81d4-46f4-9109-3654599f47c1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.445695 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442424 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-slash\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.445695 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442457 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-cni-netd\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.445695 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442459 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27f875a5-8c79-42e8-b05f-819fe5dcc452-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.445695 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442463 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-slash\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.445695 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442305 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-run-ovn-kubernetes\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.445695 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442490 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d422e92a-be96-4d4a-8011-66ff2945286d-host-cni-netd\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.445695 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442542 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0dbfe656-ce52-4013-9445-7adb45ee8fb8-cni-binary-copy\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.445695 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442592 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27f875a5-8c79-42e8-b05f-819fe5dcc452-system-cni-dir\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.445695 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442749 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/99098e94-f6ae-4918-b007-e85999dfe259-konnectivity-ca\") pod \"konnectivity-agent-6gm25\" (UID: \"99098e94-f6ae-4918-b007-e85999dfe259\") " pod="kube-system/konnectivity-agent-6gm25" Jul 06 18:58:58.445695 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442891 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-host-run-netns\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.445695 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442922 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-host-var-lib-cni-bin\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.445695 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442961 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-multus-conf-dir\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.445695 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442967 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d422e92a-be96-4d4a-8011-66ff2945286d-env-overrides\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.445695 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442985 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c996c2ca-cb6f-420e-b4d0-5b9720137b7d-serviceca\") pod \"node-ca-v2xjc\" (UID: \"c996c2ca-cb6f-420e-b4d0-5b9720137b7d\") " pod="openshift-image-registry/node-ca-v2xjc" Jul 06 18:58:58.446177 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.442993 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-host-run-netns\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.446177 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.443054 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-multus-conf-dir\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.446177 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.443094 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-host-var-lib-cni-bin\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.446177 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.443119 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/27f875a5-8c79-42e8-b05f-819fe5dcc452-os-release\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.446177 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.443209 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7kkf\" (UniqueName: \"kubernetes.io/projected/8dd572f4-3c97-402e-8480-6209afbdf600-kube-api-access-s7kkf\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.446177 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.443234 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-multus-socket-dir-parent\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.446177 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.443259 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-host-var-lib-cni-multus\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.446177 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.443284 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmgrx\" (UniqueName: \"kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx\") pod \"network-check-target-hklfk\" (UID: \"aab0c2d2-909e-466d-bd99-c4f4b3830060\") " pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:58:58.446177 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.443310 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c996c2ca-cb6f-420e-b4d0-5b9720137b7d-host\") pod \"node-ca-v2xjc\" (UID: \"c996c2ca-cb6f-420e-b4d0-5b9720137b7d\") " pod="openshift-image-registry/node-ca-v2xjc" Jul 06 18:58:58.446177 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.443372 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c996c2ca-cb6f-420e-b4d0-5b9720137b7d-serviceca\") pod \"node-ca-v2xjc\" (UID: \"c996c2ca-cb6f-420e-b4d0-5b9720137b7d\") " pod="openshift-image-registry/node-ca-v2xjc" Jul 06 18:58:58.446177 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.443377 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c996c2ca-cb6f-420e-b4d0-5b9720137b7d-host\") pod \"node-ca-v2xjc\" (UID: \"c996c2ca-cb6f-420e-b4d0-5b9720137b7d\") " pod="openshift-image-registry/node-ca-v2xjc" Jul 06 18:58:58.446177 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.443418 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-multus-socket-dir-parent\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.446177 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.443463 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0dbfe656-ce52-4013-9445-7adb45ee8fb8-host-var-lib-cni-multus\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.446177 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.443546 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/27f875a5-8c79-42e8-b05f-819fe5dcc452-os-release\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.446177 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.445125 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8dd572f4-3c97-402e-8480-6209afbdf600-tmp\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.446177 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.445206 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8dd572f4-3c97-402e-8480-6209afbdf600-etc-tuned\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.446177 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.445266 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d422e92a-be96-4d4a-8011-66ff2945286d-ovn-node-metrics-cert\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.446177 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.445545 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/99098e94-f6ae-4918-b007-e85999dfe259-agent-certs\") pod \"konnectivity-agent-6gm25\" (UID: \"99098e94-f6ae-4918-b007-e85999dfe259\") " pod="kube-system/konnectivity-agent-6gm25" Jul 06 18:58:58.451498 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.451481 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-f28sx" Jul 06 18:58:58.458550 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.458503 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 18:53:59 +0000 UTC" deadline="2028-01-08 09:06:15.296968288 +0000 UTC" Jul 06 18:58:58.458550 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.458548 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13214h7m16.838422386s" Jul 06 18:58:58.461769 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.461753 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d945k\" (UniqueName: \"kubernetes.io/projected/eb86a840-dfb9-4800-9a40-080862ec9018-kube-api-access-d945k\") pod \"iptables-alerter-842q9\" (UID: \"eb86a840-dfb9-4800-9a40-080862ec9018\") " pod="openshift-network-operator/iptables-alerter-842q9" Jul 06 18:58:58.462252 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.462228 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggl6v\" (UniqueName: \"kubernetes.io/projected/e7b6d1cb-2542-4861-9475-25365c8a5d15-kube-api-access-ggl6v\") pod \"network-metrics-daemon-jmvjq\" (UID: \"e7b6d1cb-2542-4861-9475-25365c8a5d15\") " pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:58:58.463193 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.463171 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5gck\" (UniqueName: \"kubernetes.io/projected/27f875a5-8c79-42e8-b05f-819fe5dcc452-kube-api-access-w5gck\") pod \"multus-additional-cni-plugins-vlq9c\" (UID: \"27f875a5-8c79-42e8-b05f-819fe5dcc452\") " pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.475788 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:58.475769 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 18:58:58.475788 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:58.475790 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 18:58:58.475894 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:58.475800 2553 projected.go:196] Error preparing data for projected volume kube-api-access-cmgrx for pod openshift-network-diagnostics/network-check-target-hklfk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 18:58:58.475894 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:58:58.475851 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx podName:aab0c2d2-909e-466d-bd99-c4f4b3830060 nodeName:}" failed. No retries permitted until 2026-07-06 18:58:58.975838021 +0000 UTC m=+2.033821059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cmgrx" (UniqueName: "kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx") pod "network-check-target-hklfk" (UID: "aab0c2d2-909e-466d-bd99-c4f4b3830060") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 18:58:58.477913 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.477889 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpkjh\" (UniqueName: \"kubernetes.io/projected/2f70581d-708c-43ae-871a-a126644cefff-kube-api-access-cpkjh\") pod \"node-resolver-nb9d8\" (UID: \"2f70581d-708c-43ae-871a-a126644cefff\") " pod="openshift-dns/node-resolver-nb9d8" Jul 06 18:58:58.478197 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.478176 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7kkf\" (UniqueName: \"kubernetes.io/projected/8dd572f4-3c97-402e-8480-6209afbdf600-kube-api-access-s7kkf\") pod \"tuned-fl6nt\" (UID: \"8dd572f4-3c97-402e-8480-6209afbdf600\") " pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.479125 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.479104 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqrlf\" (UniqueName: \"kubernetes.io/projected/c996c2ca-cb6f-420e-b4d0-5b9720137b7d-kube-api-access-pqrlf\") pod \"node-ca-v2xjc\" (UID: \"c996c2ca-cb6f-420e-b4d0-5b9720137b7d\") " pod="openshift-image-registry/node-ca-v2xjc" Jul 06 18:58:58.481017 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:58.480992 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e528d50850ba268ed5e671edfd2eb2.slice/crio-885c94a56e8483c0968fe5af8a29d02b7ae2e40daf9a05026db84f3b4724fd17 WatchSource:0}: Error finding container 885c94a56e8483c0968fe5af8a29d02b7ae2e40daf9a05026db84f3b4724fd17: Status 404 returned error can't find the container with id 885c94a56e8483c0968fe5af8a29d02b7ae2e40daf9a05026db84f3b4724fd17 Jul 06 18:58:58.481456 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:58.481438 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5c007a1cc8c7f01af448c594df5b2e.slice/crio-9e11f2baea5b213d14ff88838102880012349fdad5320e38ff6d8200da42807c WatchSource:0}: Error finding container 9e11f2baea5b213d14ff88838102880012349fdad5320e38ff6d8200da42807c: Status 404 returned error can't find the container with id 9e11f2baea5b213d14ff88838102880012349fdad5320e38ff6d8200da42807c Jul 06 18:58:58.483297 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.483279 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8n7g\" (UniqueName: \"kubernetes.io/projected/551e1a05-81d4-46f4-9109-3654599f47c1-kube-api-access-w8n7g\") pod \"aws-ebs-csi-driver-node-bx88g\" (UID: \"551e1a05-81d4-46f4-9109-3654599f47c1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.483376 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.483334 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5nsw\" (UniqueName: \"kubernetes.io/projected/d422e92a-be96-4d4a-8011-66ff2945286d-kube-api-access-b5nsw\") pod \"ovnkube-node-x5g5z\" (UID: \"d422e92a-be96-4d4a-8011-66ff2945286d\") " pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.487547 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.487531 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 18:58:58.491144 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.491095 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-237.ec2.internal" event={"ID":"62e528d50850ba268ed5e671edfd2eb2","Type":"ContainerStarted","Data":"885c94a56e8483c0968fe5af8a29d02b7ae2e40daf9a05026db84f3b4724fd17"} Jul 06 18:58:58.492157 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.492139 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal" event={"ID":"6f5c007a1cc8c7f01af448c594df5b2e","Type":"ContainerStarted","Data":"9e11f2baea5b213d14ff88838102880012349fdad5320e38ff6d8200da42807c"} Jul 06 18:58:58.492311 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.492296 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqz8t\" (UniqueName: \"kubernetes.io/projected/0dbfe656-ce52-4013-9445-7adb45ee8fb8-kube-api-access-sqz8t\") pod \"multus-vzw9m\" (UID: \"0dbfe656-ce52-4013-9445-7adb45ee8fb8\") " pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.515815 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.515790 2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 18:58:58.628978 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.628924 2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 18:58:58.644335 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.644313 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-842q9" Jul 06 18:58:58.651541 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:58.651522 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb86a840_dfb9_4800_9a40_080862ec9018.slice/crio-49e69e1100b6e60f04e45059c5cde8a0ddd9ae5fa94103ab962f540264c78daf WatchSource:0}: Error finding container 49e69e1100b6e60f04e45059c5cde8a0ddd9ae5fa94103ab962f540264c78daf: Status 404 returned error can't find the container with id 49e69e1100b6e60f04e45059c5cde8a0ddd9ae5fa94103ab962f540264c78daf Jul 06 18:58:58.663632 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.663596 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" Jul 06 18:58:58.670761 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:58.670743 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551e1a05_81d4_46f4_9109_3654599f47c1.slice/crio-b0ecfd70f03fea88b1f58b76adb271cae51aa74091ab319ddae8598a00ff6ff0 WatchSource:0}: Error finding container b0ecfd70f03fea88b1f58b76adb271cae51aa74091ab319ddae8598a00ff6ff0: Status 404 returned error can't find the container with id b0ecfd70f03fea88b1f58b76adb271cae51aa74091ab319ddae8598a00ff6ff0 Jul 06 18:58:58.683064 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.683047 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" Jul 06 18:58:58.685562 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.685547 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v2xjc" Jul 06 18:58:58.690079 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:58.690059 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd572f4_3c97_402e_8480_6209afbdf600.slice/crio-209a9f03a778e6118f1e92fff2b76052d298a355ff70ed2f44a8021bf7f3eb8f WatchSource:0}: Error finding container 209a9f03a778e6118f1e92fff2b76052d298a355ff70ed2f44a8021bf7f3eb8f: Status 404 returned error can't find the container with id 209a9f03a778e6118f1e92fff2b76052d298a355ff70ed2f44a8021bf7f3eb8f Jul 06 18:58:58.692746 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:58.692724 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc996c2ca_cb6f_420e_b4d0_5b9720137b7d.slice/crio-6a4ff7014c6e89dfa21af093ebe524cea16bf56f2eea280793eb6f8fd3a115da WatchSource:0}: Error finding container 6a4ff7014c6e89dfa21af093ebe524cea16bf56f2eea280793eb6f8fd3a115da: Status 404 returned error can't find the container with id 6a4ff7014c6e89dfa21af093ebe524cea16bf56f2eea280793eb6f8fd3a115da Jul 06 18:58:58.704664 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.704626 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:58:58.710645 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.710624 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6gm25" Jul 06 18:58:58.711569 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:58.711551 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd422e92a_be96_4d4a_8011_66ff2945286d.slice/crio-010f798eb0157b551a235862bb9e0e89468014326c3b65c0c630a4e9262a7200 WatchSource:0}: Error finding container 010f798eb0157b551a235862bb9e0e89468014326c3b65c0c630a4e9262a7200: Status 404 returned error can't find the container with id 010f798eb0157b551a235862bb9e0e89468014326c3b65c0c630a4e9262a7200 Jul 06 18:58:58.726542 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.726528 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nb9d8" Jul 06 18:58:58.733643 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:58.733606 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f70581d_708c_43ae_871a_a126644cefff.slice/crio-eb0453693a50c2dbeb7258230ec073c676eebb4a814ce840b97c03ec4c34f24e WatchSource:0}: Error finding container eb0453693a50c2dbeb7258230ec073c676eebb4a814ce840b97c03ec4c34f24e: Status 404 returned error can't find the container with id eb0453693a50c2dbeb7258230ec073c676eebb4a814ce840b97c03ec4c34f24e Jul 06 18:58:58.739267 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.739250 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vlq9c" Jul 06 18:58:58.746992 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:58.746972 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f875a5_8c79_42e8_b05f_819fe5dcc452.slice/crio-5f753463713f6a31dfd36899f80c28d18eb8f7f637bd772efc219765d6f3d878 WatchSource:0}: Error finding container 5f753463713f6a31dfd36899f80c28d18eb8f7f637bd772efc219765d6f3d878: Status 404 returned error can't find the container with id 5f753463713f6a31dfd36899f80c28d18eb8f7f637bd772efc219765d6f3d878 Jul 06 18:58:58.752785 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:58:58.752769 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vzw9m" Jul 06 18:58:58.759500 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:58:58.759483 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbfe656_ce52_4013_9445_7adb45ee8fb8.slice/crio-2dc3714ee6a7b9b018136780c064ed9edbf207d9ccd0c08e22dccb3306da02f4 WatchSource:0}: Error finding container 2dc3714ee6a7b9b018136780c064ed9edbf207d9ccd0c08e22dccb3306da02f4: Status 404 returned error can't find the container with id 2dc3714ee6a7b9b018136780c064ed9edbf207d9ccd0c08e22dccb3306da02f4 Jul 06 18:59:00.735174 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:00.735153 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs\") pod \"network-metrics-daemon-jmvjq\" (UID: \"e7b6d1cb-2542-4861-9475-25365c8a5d15\") " pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:00.735257 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:00.735187 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret\") pod \"global-pull-secret-syncer-v5xs7\" (UID: \"f7d2aeda-aa6f-4865-ade0-f9aa24ea785a\") " pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:00.735314 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:00.735296 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 18:59:00.735314 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:00.735296 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 18:59:00.735395 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:00.735358 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret podName:f7d2aeda-aa6f-4865-ade0-f9aa24ea785a nodeName:}" failed. No retries permitted until 2026-07-06 18:59:01.735340695 +0000 UTC m=+3.004523030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret") pod "global-pull-secret-syncer-v5xs7" (UID: "f7d2aeda-aa6f-4865-ade0-f9aa24ea785a") : object "kube-system"/"original-pull-secret" not registered Jul 06 18:59:00.735395 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:00.735371 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs podName:e7b6d1cb-2542-4861-9475-25365c8a5d15 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:01.735365636 +0000 UTC m=+3.004547955 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs") pod "network-metrics-daemon-jmvjq" (UID: "e7b6d1cb-2542-4861-9475-25365c8a5d15") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 18:59:00.835656 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:00.835622 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmgrx\" (UniqueName: \"kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx\") pod \"network-check-target-hklfk\" (UID: \"aab0c2d2-909e-466d-bd99-c4f4b3830060\") " pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:00.835825 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:00.835805 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 18:59:00.835877 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:00.835834 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 18:59:00.835877 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:00.835847 2553 projected.go:196] Error preparing data for projected volume kube-api-access-cmgrx for pod openshift-network-diagnostics/network-check-target-hklfk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 18:59:00.835965 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:00.835915 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx podName:aab0c2d2-909e-466d-bd99-c4f4b3830060 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:01.835889722 +0000 UTC m=+3.105072054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cmgrx" (UniqueName: "kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx") pod "network-check-target-hklfk" (UID: "aab0c2d2-909e-466d-bd99-c4f4b3830060") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 18:59:01.242755 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:01.242704 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 18:54:00 +0000 UTC" deadline="2027-12-29 10:59:10.788772885 +0000 UTC" Jul 06 18:59:01.242755 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:01.242754 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12976h0m9.546024302s" Jul 06 18:59:01.277576 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:01.277532 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:01.277743 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:01.277670 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5xs7" podUID="f7d2aeda-aa6f-4865-ade0-f9aa24ea785a" Jul 06 18:59:01.277743 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:01.277714 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:01.277861 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:01.277842 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmvjq" podUID="e7b6d1cb-2542-4861-9475-25365c8a5d15" Jul 06 18:59:01.307089 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:01.306989 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v2xjc" event={"ID":"c996c2ca-cb6f-420e-b4d0-5b9720137b7d","Type":"ContainerStarted","Data":"6a4ff7014c6e89dfa21af093ebe524cea16bf56f2eea280793eb6f8fd3a115da"} Jul 06 18:59:01.318307 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:01.318109 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-842q9" event={"ID":"eb86a840-dfb9-4800-9a40-080862ec9018","Type":"ContainerStarted","Data":"49e69e1100b6e60f04e45059c5cde8a0ddd9ae5fa94103ab962f540264c78daf"} Jul 06 18:59:01.333238 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:01.333205 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6gm25" event={"ID":"99098e94-f6ae-4918-b007-e85999dfe259","Type":"ContainerStarted","Data":"84e10cf1fe6e81fffcf36c650687bf27ec5c8c84fafa77e522e99d3a6e9fb482"} Jul 06 18:59:01.349626 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:01.349593 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" event={"ID":"8dd572f4-3c97-402e-8480-6209afbdf600","Type":"ContainerStarted","Data":"209a9f03a778e6118f1e92fff2b76052d298a355ff70ed2f44a8021bf7f3eb8f"} Jul 06 18:59:01.372088 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:01.372015 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" event={"ID":"551e1a05-81d4-46f4-9109-3654599f47c1","Type":"ContainerStarted","Data":"b0ecfd70f03fea88b1f58b76adb271cae51aa74091ab319ddae8598a00ff6ff0"} Jul 06 18:59:01.402657 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:01.402621 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vzw9m" event={"ID":"0dbfe656-ce52-4013-9445-7adb45ee8fb8","Type":"ContainerStarted","Data":"2dc3714ee6a7b9b018136780c064ed9edbf207d9ccd0c08e22dccb3306da02f4"} Jul 06 18:59:01.431437 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:01.427571 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlq9c" event={"ID":"27f875a5-8c79-42e8-b05f-819fe5dcc452","Type":"ContainerStarted","Data":"5f753463713f6a31dfd36899f80c28d18eb8f7f637bd772efc219765d6f3d878"} Jul 06 18:59:01.446453 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:01.446221 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nb9d8" event={"ID":"2f70581d-708c-43ae-871a-a126644cefff","Type":"ContainerStarted","Data":"eb0453693a50c2dbeb7258230ec073c676eebb4a814ce840b97c03ec4c34f24e"} Jul 06 18:59:01.460780 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:01.460755 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 18:59:01.478435 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:01.474763 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" event={"ID":"d422e92a-be96-4d4a-8011-66ff2945286d","Type":"ContainerStarted","Data":"010f798eb0157b551a235862bb9e0e89468014326c3b65c0c630a4e9262a7200"} Jul 06 18:59:01.743012 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:01.742976 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs\") pod \"network-metrics-daemon-jmvjq\" (UID: \"e7b6d1cb-2542-4861-9475-25365c8a5d15\") " pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:01.743194 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:01.743036 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret\") pod \"global-pull-secret-syncer-v5xs7\" (UID: \"f7d2aeda-aa6f-4865-ade0-f9aa24ea785a\") " pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:01.743282 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:01.743189 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 18:59:01.743282 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:01.743253 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret podName:f7d2aeda-aa6f-4865-ade0-f9aa24ea785a nodeName:}" failed. No retries permitted until 2026-07-06 18:59:03.743235211 +0000 UTC m=+5.012417546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret") pod "global-pull-secret-syncer-v5xs7" (UID: "f7d2aeda-aa6f-4865-ade0-f9aa24ea785a") : object "kube-system"/"original-pull-secret" not registered Jul 06 18:59:01.743460 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:01.743322 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 18:59:01.743460 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:01.743356 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs podName:e7b6d1cb-2542-4861-9475-25365c8a5d15 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:03.743344346 +0000 UTC m=+5.012526680 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs") pod "network-metrics-daemon-jmvjq" (UID: "e7b6d1cb-2542-4861-9475-25365c8a5d15") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 18:59:01.844323 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:01.844287 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmgrx\" (UniqueName: \"kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx\") pod \"network-check-target-hklfk\" (UID: \"aab0c2d2-909e-466d-bd99-c4f4b3830060\") " pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:01.844566 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:01.844516 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 18:59:01.844566 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:01.844539 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 18:59:01.844566 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:01.844552 2553 projected.go:196] Error preparing data for projected volume kube-api-access-cmgrx for pod openshift-network-diagnostics/network-check-target-hklfk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 18:59:01.844739 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:01.844609 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx podName:aab0c2d2-909e-466d-bd99-c4f4b3830060 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:03.844591745 +0000 UTC m=+5.113774075 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cmgrx" (UniqueName: "kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx") pod "network-check-target-hklfk" (UID: "aab0c2d2-909e-466d-bd99-c4f4b3830060") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 18:59:02.243105 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:02.243033 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 18:54:00 +0000 UTC" deadline="2027-12-30 09:08:33.06328517 +0000 UTC" Jul 06 18:59:02.243105 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:02.243086 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12998h9m30.820201162s" Jul 06 18:59:02.276957 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:02.276902 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:02.277111 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:02.277080 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hklfk" podUID="aab0c2d2-909e-466d-bd99-c4f4b3830060" Jul 06 18:59:03.277858 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:03.277552 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:03.278318 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:03.277904 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5xs7" podUID="f7d2aeda-aa6f-4865-ade0-f9aa24ea785a" Jul 06 18:59:03.278318 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:03.278289 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:03.278458 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:03.278391 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmvjq" podUID="e7b6d1cb-2542-4861-9475-25365c8a5d15" Jul 06 18:59:03.763171 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:03.763138 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs\") pod \"network-metrics-daemon-jmvjq\" (UID: \"e7b6d1cb-2542-4861-9475-25365c8a5d15\") " pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:03.763350 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:03.763189 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret\") pod \"global-pull-secret-syncer-v5xs7\" (UID: \"f7d2aeda-aa6f-4865-ade0-f9aa24ea785a\") " pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:03.763350 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:03.763322 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 18:59:03.763486 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:03.763378 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret podName:f7d2aeda-aa6f-4865-ade0-f9aa24ea785a nodeName:}" failed. No retries permitted until 2026-07-06 18:59:07.763360171 +0000 UTC m=+9.032542510 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret") pod "global-pull-secret-syncer-v5xs7" (UID: "f7d2aeda-aa6f-4865-ade0-f9aa24ea785a") : object "kube-system"/"original-pull-secret" not registered Jul 06 18:59:03.763790 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:03.763772 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 18:59:03.763857 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:03.763826 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs podName:e7b6d1cb-2542-4861-9475-25365c8a5d15 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:07.763811614 +0000 UTC m=+9.032993938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs") pod "network-metrics-daemon-jmvjq" (UID: "e7b6d1cb-2542-4861-9475-25365c8a5d15") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 18:59:03.864354 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:03.864320 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmgrx\" (UniqueName: \"kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx\") pod \"network-check-target-hklfk\" (UID: \"aab0c2d2-909e-466d-bd99-c4f4b3830060\") " pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:03.864526 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:03.864491 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 18:59:03.864526 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:03.864514 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 18:59:03.864526 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:03.864527 2553 projected.go:196] Error preparing data for projected volume kube-api-access-cmgrx for pod openshift-network-diagnostics/network-check-target-hklfk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 18:59:03.864675 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:03.864580 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx podName:aab0c2d2-909e-466d-bd99-c4f4b3830060 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:07.864562059 +0000 UTC m=+9.133744399 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cmgrx" (UniqueName: "kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx") pod "network-check-target-hklfk" (UID: "aab0c2d2-909e-466d-bd99-c4f4b3830060") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 18:59:04.277055 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:04.276583 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:04.277055 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:04.276713 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hklfk" podUID="aab0c2d2-909e-466d-bd99-c4f4b3830060" Jul 06 18:59:05.278892 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:05.276647 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:05.278892 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:05.276782 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5xs7" podUID="f7d2aeda-aa6f-4865-ade0-f9aa24ea785a" Jul 06 18:59:05.278892 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:05.277240 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:05.278892 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:05.277356 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmvjq" podUID="e7b6d1cb-2542-4861-9475-25365c8a5d15" Jul 06 18:59:06.277351 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:06.277313 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:06.277522 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:06.277468 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hklfk" podUID="aab0c2d2-909e-466d-bd99-c4f4b3830060" Jul 06 18:59:07.278568 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:07.278533 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:07.279013 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:07.278663 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5xs7" podUID="f7d2aeda-aa6f-4865-ade0-f9aa24ea785a" Jul 06 18:59:07.279874 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:07.279856 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:07.279976 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:07.279962 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmvjq" podUID="e7b6d1cb-2542-4861-9475-25365c8a5d15" Jul 06 18:59:07.795299 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:07.795267 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs\") pod \"network-metrics-daemon-jmvjq\" (UID: \"e7b6d1cb-2542-4861-9475-25365c8a5d15\") " pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:07.795519 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:07.795316 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret\") pod \"global-pull-secret-syncer-v5xs7\" (UID: \"f7d2aeda-aa6f-4865-ade0-f9aa24ea785a\") " pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:07.795519 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:07.795448 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 18:59:07.795519 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:07.795489 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 18:59:07.795519 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:07.795516 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs podName:e7b6d1cb-2542-4861-9475-25365c8a5d15 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:15.795495206 +0000 UTC m=+17.064677549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs") pod "network-metrics-daemon-jmvjq" (UID: "e7b6d1cb-2542-4861-9475-25365c8a5d15") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 18:59:07.795734 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:07.795540 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret podName:f7d2aeda-aa6f-4865-ade0-f9aa24ea785a nodeName:}" failed. No retries permitted until 2026-07-06 18:59:15.79552717 +0000 UTC m=+17.064709489 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret") pod "global-pull-secret-syncer-v5xs7" (UID: "f7d2aeda-aa6f-4865-ade0-f9aa24ea785a") : object "kube-system"/"original-pull-secret" not registered Jul 06 18:59:07.896346 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:07.896307 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmgrx\" (UniqueName: \"kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx\") pod \"network-check-target-hklfk\" (UID: \"aab0c2d2-909e-466d-bd99-c4f4b3830060\") " pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:07.896517 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:07.896474 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 18:59:07.896517 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:07.896492 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 18:59:07.896517 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:07.896505 2553 projected.go:196] Error preparing data for projected volume kube-api-access-cmgrx for pod openshift-network-diagnostics/network-check-target-hklfk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 18:59:07.896649 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:07.896561 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx podName:aab0c2d2-909e-466d-bd99-c4f4b3830060 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:15.896540789 +0000 UTC m=+17.165723109 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cmgrx" (UniqueName: "kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx") pod "network-check-target-hklfk" (UID: "aab0c2d2-909e-466d-bd99-c4f4b3830060") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 18:59:08.277363 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:08.277327 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:08.277569 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:08.277462 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hklfk" podUID="aab0c2d2-909e-466d-bd99-c4f4b3830060" Jul 06 18:59:09.278183 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:09.278156 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:09.278586 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:09.278167 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:09.278586 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:09.278257 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmvjq" podUID="e7b6d1cb-2542-4861-9475-25365c8a5d15" Jul 06 18:59:09.278586 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:09.278328 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5xs7" podUID="f7d2aeda-aa6f-4865-ade0-f9aa24ea785a" Jul 06 18:59:10.276956 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:10.276921 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:10.277128 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:10.277034 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hklfk" podUID="aab0c2d2-909e-466d-bd99-c4f4b3830060" Jul 06 18:59:11.276809 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:11.276775 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:11.277237 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:11.276775 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:11.277237 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:11.276901 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5xs7" podUID="f7d2aeda-aa6f-4865-ade0-f9aa24ea785a" Jul 06 18:59:11.277237 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:11.276988 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmvjq" podUID="e7b6d1cb-2542-4861-9475-25365c8a5d15" Jul 06 18:59:12.276948 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:12.276923 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:12.277304 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:12.277014 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hklfk" podUID="aab0c2d2-909e-466d-bd99-c4f4b3830060" Jul 06 18:59:13.277434 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:13.277386 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:13.277876 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:13.277386 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:13.277876 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:13.277531 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5xs7" podUID="f7d2aeda-aa6f-4865-ade0-f9aa24ea785a" Jul 06 18:59:13.277876 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:13.277648 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmvjq" podUID="e7b6d1cb-2542-4861-9475-25365c8a5d15" Jul 06 18:59:14.277381 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:14.277353 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:14.277547 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:14.277464 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hklfk" podUID="aab0c2d2-909e-466d-bd99-c4f4b3830060" Jul 06 18:59:15.276783 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:15.276747 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:15.276956 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:15.276874 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5xs7" podUID="f7d2aeda-aa6f-4865-ade0-f9aa24ea785a" Jul 06 18:59:15.276956 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:15.276938 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:15.277099 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:15.277066 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmvjq" podUID="e7b6d1cb-2542-4861-9475-25365c8a5d15" Jul 06 18:59:15.853900 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:15.853868 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs\") pod \"network-metrics-daemon-jmvjq\" (UID: \"e7b6d1cb-2542-4861-9475-25365c8a5d15\") " pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:15.853900 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:15.853910 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret\") pod \"global-pull-secret-syncer-v5xs7\" (UID: \"f7d2aeda-aa6f-4865-ade0-f9aa24ea785a\") " pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:15.854447 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:15.854047 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 18:59:15.854447 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:15.854048 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 18:59:15.854447 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:15.854106 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret podName:f7d2aeda-aa6f-4865-ade0-f9aa24ea785a nodeName:}" failed. No retries permitted until 2026-07-06 18:59:31.854088241 +0000 UTC m=+33.123270569 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret") pod "global-pull-secret-syncer-v5xs7" (UID: "f7d2aeda-aa6f-4865-ade0-f9aa24ea785a") : object "kube-system"/"original-pull-secret" not registered Jul 06 18:59:15.854447 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:15.854125 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs podName:e7b6d1cb-2542-4861-9475-25365c8a5d15 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:31.854115609 +0000 UTC m=+33.123297934 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs") pod "network-metrics-daemon-jmvjq" (UID: "e7b6d1cb-2542-4861-9475-25365c8a5d15") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 18:59:15.955037 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:15.955003 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmgrx\" (UniqueName: \"kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx\") pod \"network-check-target-hklfk\" (UID: \"aab0c2d2-909e-466d-bd99-c4f4b3830060\") " pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:15.955216 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:15.955118 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 18:59:15.955216 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:15.955131 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 18:59:15.955216 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:15.955142 2553 projected.go:196] Error preparing data for projected volume kube-api-access-cmgrx for pod openshift-network-diagnostics/network-check-target-hklfk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 18:59:15.955216 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:15.955205 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx podName:aab0c2d2-909e-466d-bd99-c4f4b3830060 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:31.955189767 +0000 UTC m=+33.224372110 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cmgrx" (UniqueName: "kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx") pod "network-check-target-hklfk" (UID: "aab0c2d2-909e-466d-bd99-c4f4b3830060") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 18:59:16.276782 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:16.276742 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:16.276971 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:16.276871 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hklfk" podUID="aab0c2d2-909e-466d-bd99-c4f4b3830060" Jul 06 18:59:17.277064 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:17.277027 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:17.277507 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:17.277166 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5xs7" podUID="f7d2aeda-aa6f-4865-ade0-f9aa24ea785a" Jul 06 18:59:17.277507 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:17.277208 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:17.277507 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:17.277321 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmvjq" podUID="e7b6d1cb-2542-4861-9475-25365c8a5d15" Jul 06 18:59:18.277208 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:18.277175 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:18.277657 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:18.277290 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hklfk" podUID="aab0c2d2-909e-466d-bd99-c4f4b3830060" Jul 06 18:59:19.277274 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:19.277239 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:19.277792 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:19.277364 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmvjq" podUID="e7b6d1cb-2542-4861-9475-25365c8a5d15" Jul 06 18:59:19.277792 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:19.277432 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:19.277792 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:19.277543 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5xs7" podUID="f7d2aeda-aa6f-4865-ade0-f9aa24ea785a" Jul 06 18:59:20.277435 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:20.277391 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:20.277846 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:20.277516 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hklfk" podUID="aab0c2d2-909e-466d-bd99-c4f4b3830060" Jul 06 18:59:21.277157 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:21.276964 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:21.277290 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:21.276992 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:21.277290 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:21.277270 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmvjq" podUID="e7b6d1cb-2542-4861-9475-25365c8a5d15" Jul 06 18:59:21.277499 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:21.277434 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5xs7" podUID="f7d2aeda-aa6f-4865-ade0-f9aa24ea785a" Jul 06 18:59:21.519032 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:21.518972 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" event={"ID":"8dd572f4-3c97-402e-8480-6209afbdf600","Type":"ContainerStarted","Data":"1f3e6a2eac3658d9d89632d3e90e9efc2cf5166ff48a6139fd716ad4b7cb2c26"} Jul 06 18:59:21.520954 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:21.520925 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vzw9m" event={"ID":"0dbfe656-ce52-4013-9445-7adb45ee8fb8","Type":"ContainerStarted","Data":"c393733e063a2f96570ff83e03b65852efe3061e84b67ad3ea26140cad2d0ace"} Jul 06 18:59:21.524108 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:21.523979 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 18:59:21.524476 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:21.524373 2553 generic.go:356] "Generic (PLEG): container finished" podID="d422e92a-be96-4d4a-8011-66ff2945286d" containerID="560d8702438f037a7dc69647227d6f9aa892fae245ccec3d227234632f4de8a1" exitCode=1 Jul 06 18:59:21.524552 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:21.524486 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" event={"ID":"d422e92a-be96-4d4a-8011-66ff2945286d","Type":"ContainerStarted","Data":"4630b88e813d7decf21ca04377cb8dfa61240f0e45ae265c21d9bc86e83ba48b"} Jul 06 18:59:21.524600 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:21.524549 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" event={"ID":"d422e92a-be96-4d4a-8011-66ff2945286d","Type":"ContainerStarted","Data":"0f623cfff792e3eda65e32d36236a254e42fb07c07381ec5941ed0e02db3cb43"} Jul 06 18:59:21.524642 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:21.524598 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" event={"ID":"d422e92a-be96-4d4a-8011-66ff2945286d","Type":"ContainerStarted","Data":"6ec3c7096abc52d45cd7f201422a31dfd891824cdfad057907816c8f577b6843"} Jul 06 18:59:21.524685 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:21.524639 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" event={"ID":"d422e92a-be96-4d4a-8011-66ff2945286d","Type":"ContainerStarted","Data":"1d4f3e6aaac7842ef399df3f4d916049220cca369e10afa8ac8b8abf298bff33"} Jul 06 18:59:21.524728 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:21.524685 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" event={"ID":"d422e92a-be96-4d4a-8011-66ff2945286d","Type":"ContainerDied","Data":"560d8702438f037a7dc69647227d6f9aa892fae245ccec3d227234632f4de8a1"} Jul 06 18:59:21.524761 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:21.524734 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" event={"ID":"d422e92a-be96-4d4a-8011-66ff2945286d","Type":"ContainerStarted","Data":"94918f617bafffef7828b911ead5237c076a696901efed5b87b8fd7fd3397c87"} Jul 06 18:59:21.525881 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:21.525858 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-237.ec2.internal" event={"ID":"62e528d50850ba268ed5e671edfd2eb2","Type":"ContainerStarted","Data":"d7d7e6ab18910637c1d28b2fd9b2d2c649adfc96eb5f47c7cc49c92e7bf594b6"} Jul 06 18:59:21.539371 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:21.539331 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fl6nt" podStartSLOduration=2.367200332 podStartE2EDuration="22.53932s" totalImagesPullingTime="20.172119668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:59:21.538691191 +0000 UTC m=+22.807873533" watchObservedRunningTime="2026-07-06 18:59:21.53932 +0000 UTC m=+22.808502341" Jul 06 18:59:21.579335 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:21.579292 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-237.ec2.internal" podStartSLOduration=21.579280048 podStartE2EDuration="21.579280048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:59:21.55660145 +0000 UTC m=+22.825783791" watchObservedRunningTime="2026-07-06 18:59:21.579280048 +0000 UTC m=+22.848462391" Jul 06 18:59:21.579551 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:21.579528 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vzw9m" podStartSLOduration=2.421650665 podStartE2EDuration="22.579522601s" totalImagesPullingTime="20.157871936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:59:21.579411273 +0000 UTC m=+22.848593615" watchObservedRunningTime="2026-07-06 18:59:21.579522601 +0000 UTC m=+22.848704944" Jul 06 18:59:22.277126 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:22.276951 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:22.277245 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:22.277197 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hklfk" podUID="aab0c2d2-909e-466d-bd99-c4f4b3830060" Jul 06 18:59:22.529480 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:22.529447 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v2xjc" event={"ID":"c996c2ca-cb6f-420e-b4d0-5b9720137b7d","Type":"ContainerStarted","Data":"dbb4ed102db7d5303785bcff546a6843193ccc9a277d68d809f088fda17fb4f9"} Jul 06 18:59:22.531147 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:22.531118 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-842q9" event={"ID":"eb86a840-dfb9-4800-9a40-080862ec9018","Type":"ContainerStarted","Data":"90987763c87eb753d0d6d183a22911bb3a6274c3797f6b82795a343cc9829a89"} Jul 06 18:59:22.532917 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:22.532879 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6gm25" event={"ID":"99098e94-f6ae-4918-b007-e85999dfe259","Type":"ContainerStarted","Data":"1249e51e302889d46dd4d6ea8164d68a6eb7a7fc0036e5d43bdb4c4840746dda"} Jul 06 18:59:22.534523 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:22.534505 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" event={"ID":"551e1a05-81d4-46f4-9109-3654599f47c1","Type":"ContainerStarted","Data":"ce9d10338ebb55dd9299d47ca2f797863608b2c820f16123d2fc9f91d5d11575"} Jul 06 18:59:22.536257 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:22.536235 2553 generic.go:356] "Generic (PLEG): container finished" podID="6f5c007a1cc8c7f01af448c594df5b2e" containerID="372a26008bf2ab1250565f1cd39322b8e3945fd14fe0c703fdce1367d70c30a3" exitCode=0 Jul 06 18:59:22.536346 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:22.536325 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal" event={"ID":"6f5c007a1cc8c7f01af448c594df5b2e","Type":"ContainerDied","Data":"372a26008bf2ab1250565f1cd39322b8e3945fd14fe0c703fdce1367d70c30a3"} Jul 06 18:59:22.538559 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:22.538437 2553 generic.go:356] "Generic (PLEG): container finished" podID="27f875a5-8c79-42e8-b05f-819fe5dcc452" containerID="84b2951cb753899203cebc8dd0150ae95bea56ae4c5f49be6dea3b1abadaa3dd" exitCode=0 Jul 06 18:59:22.538559 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:22.538516 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlq9c" event={"ID":"27f875a5-8c79-42e8-b05f-819fe5dcc452","Type":"ContainerDied","Data":"84b2951cb753899203cebc8dd0150ae95bea56ae4c5f49be6dea3b1abadaa3dd"} Jul 06 18:59:22.542557 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:22.541843 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nb9d8" event={"ID":"2f70581d-708c-43ae-871a-a126644cefff","Type":"ContainerStarted","Data":"345823c40779ee5e4b8c96f2a09b1e9d3727bfa056f9a9bec0518662f7ede0a8"} Jul 06 18:59:22.546434 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:22.546379 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v2xjc" podStartSLOduration=3.370756653 podStartE2EDuration="23.546367395s" totalImagesPullingTime="20.175610742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:59:22.546284463 +0000 UTC m=+23.815466805" watchObservedRunningTime="2026-07-06 18:59:22.546367395 +0000 UTC m=+23.815549736" Jul 06 18:59:22.564690 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:22.564647 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6gm25" podStartSLOduration=3.416570615 podStartE2EDuration="23.564635907s" totalImagesPullingTime="20.148065292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:59:22.564589464 +0000 UTC m=+23.833771805" watchObservedRunningTime="2026-07-06 18:59:22.564635907 +0000 UTC m=+23.833818249" Jul 06 18:59:22.635759 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:22.635717 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nb9d8" podStartSLOduration=3.502474574 podStartE2EDuration="23.635702471s" totalImagesPullingTime="20.133227897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:59:22.607677476 +0000 UTC m=+23.876859819" watchObservedRunningTime="2026-07-06 18:59:22.635702471 +0000 UTC m=+23.904884807" Jul 06 18:59:22.651672 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:22.651637 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-842q9" podStartSLOduration=3.429284627 podStartE2EDuration="23.651623581s" totalImagesPullingTime="20.222338954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:59:22.651564515 +0000 UTC m=+23.920746858" watchObservedRunningTime="2026-07-06 18:59:22.651623581 +0000 UTC m=+23.920805923" Jul 06 18:59:23.262710 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:23.262690 2553 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 18:59:23.276679 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:23.276656 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:23.276787 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:23.276764 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5xs7" podUID="f7d2aeda-aa6f-4865-ade0-f9aa24ea785a" Jul 06 18:59:23.276835 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:23.276803 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:23.276906 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:23.276889 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmvjq" podUID="e7b6d1cb-2542-4861-9475-25365c8a5d15" Jul 06 18:59:23.546276 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:23.546244 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal" event={"ID":"6f5c007a1cc8c7f01af448c594df5b2e","Type":"ContainerStarted","Data":"4a47e043961d47f5e868b3d5eb41d8c175e498f7cfcbb00869ba0eadd7bb023f"} Jul 06 18:59:23.548212 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:23.548187 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" event={"ID":"551e1a05-81d4-46f4-9109-3654599f47c1","Type":"ContainerStarted","Data":"3b5e5c6a9f53733db234451332e7e211199c95700fafadd9ba865595ba3a15fb"} Jul 06 18:59:24.192660 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:24.192570 2553 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T18:59:23.262706577Z","UUID":"81a34fbb-ec67-4832-82be-91bcd0271334","Handler":null,"Name":"","Endpoint":""} Jul 06 18:59:24.194330 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:24.194301 2553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 18:59:24.194330 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:24.194336 2553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 18:59:24.277276 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:24.277216 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:24.277360 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:24.277340 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hklfk" podUID="aab0c2d2-909e-466d-bd99-c4f4b3830060" Jul 06 18:59:24.552794 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:24.552571 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" event={"ID":"551e1a05-81d4-46f4-9109-3654599f47c1","Type":"ContainerStarted","Data":"2d6f247b319c4259b167da43ddc8fe85a410db961c34f57e95d7412b17cb1f23"} Jul 06 18:59:24.556111 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:24.556091 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 18:59:24.556538 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:24.556513 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" event={"ID":"d422e92a-be96-4d4a-8011-66ff2945286d","Type":"ContainerStarted","Data":"1fb7fa0254cc215e53a6a17f04ccf5f98c546aacb1b2823b9c433551d053fd50"} Jul 06 18:59:24.575649 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:24.575603 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bx88g" podStartSLOduration=3.043260744 podStartE2EDuration="25.575590474s" totalImagesPullingTime="22.53232973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:58:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:59:24.575203854 +0000 UTC m=+25.844386187" watchObservedRunningTime="2026-07-06 18:59:24.575590474 +0000 UTC m=+25.844772816" Jul 06 18:59:24.575954 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:24.575927 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-237.ec2.internal" podStartSLOduration=24.575920019 podStartE2EDuration="24.575920019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:59:23.588717101 +0000 UTC m=+24.857899442" watchObservedRunningTime="2026-07-06 18:59:24.575920019 +0000 UTC m=+25.845102381" Jul 06 18:59:24.720816 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:24.720783 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6gm25" Jul 06 18:59:24.721269 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:24.721255 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6gm25" Jul 06 18:59:25.277042 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:25.277017 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:25.277042 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:25.277049 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:25.277296 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:25.277153 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmvjq" podUID="e7b6d1cb-2542-4861-9475-25365c8a5d15" Jul 06 18:59:25.277344 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:25.277282 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5xs7" podUID="f7d2aeda-aa6f-4865-ade0-f9aa24ea785a" Jul 06 18:59:25.559262 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:25.559181 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6gm25" Jul 06 18:59:25.559703 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:25.559671 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6gm25" Jul 06 18:59:26.277319 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:26.277291 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:26.277492 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:26.277393 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hklfk" podUID="aab0c2d2-909e-466d-bd99-c4f4b3830060" Jul 06 18:59:26.564595 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:26.564519 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 18:59:26.565341 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:26.564882 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" event={"ID":"d422e92a-be96-4d4a-8011-66ff2945286d","Type":"ContainerStarted","Data":"23e3ff01369307270827349bc4ec6c35fc44107f8362a735648582efc42f1696"} Jul 06 18:59:26.565464 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:26.565448 2553 scope.go:117] "RemoveContainer" containerID="560d8702438f037a7dc69647227d6f9aa892fae245ccec3d227234632f4de8a1" Jul 06 18:59:27.277192 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:27.276989 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:27.277356 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:27.277086 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:27.277356 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:27.277316 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5xs7" podUID="f7d2aeda-aa6f-4865-ade0-f9aa24ea785a" Jul 06 18:59:27.277499 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:27.277383 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmvjq" podUID="e7b6d1cb-2542-4861-9475-25365c8a5d15" Jul 06 18:59:27.572012 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:27.571885 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 18:59:27.573116 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:27.572985 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" event={"ID":"d422e92a-be96-4d4a-8011-66ff2945286d","Type":"ContainerStarted","Data":"3cd7c6dd02f7a322bb18ae78f8a74bc6e51383bbf82f760d01ebbd9176babe4d"} Jul 06 18:59:27.573291 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:27.573270 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:59:27.573369 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:27.573356 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:59:27.573461 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:27.573407 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:59:27.591081 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:27.591053 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:59:27.591764 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:27.591742 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 18:59:27.622006 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:27.621711 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" podStartSLOduration=8.397972163 podStartE2EDuration="28.621692139s" totalImagesPullingTime="20.223719976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:59:27.621622783 +0000 UTC m=+28.890805127" watchObservedRunningTime="2026-07-06 18:59:27.621692139 +0000 UTC m=+28.890874481" Jul 06 18:59:28.046268 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:28.046228 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v5xs7"] Jul 06 18:59:28.046454 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:28.046357 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:28.046538 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:28.046492 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5xs7" podUID="f7d2aeda-aa6f-4865-ade0-f9aa24ea785a" Jul 06 18:59:28.049588 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:28.048889 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hklfk"] Jul 06 18:59:28.049588 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:28.048993 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:28.049588 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:28.049076 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hklfk" podUID="aab0c2d2-909e-466d-bd99-c4f4b3830060" Jul 06 18:59:28.071114 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:28.071064 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jmvjq"] Jul 06 18:59:28.071303 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:28.071174 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:28.071303 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:28.071275 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmvjq" podUID="e7b6d1cb-2542-4861-9475-25365c8a5d15" Jul 06 18:59:29.277814 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:29.277781 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:29.278171 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:29.277882 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmvjq" podUID="e7b6d1cb-2542-4861-9475-25365c8a5d15" Jul 06 18:59:29.579803 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:29.579765 2553 generic.go:356] "Generic (PLEG): container finished" podID="27f875a5-8c79-42e8-b05f-819fe5dcc452" containerID="4bd37f76e39e53c6bb8c7e60f84f8c5e6673951c72cb7d8ddd24cddd45491d84" exitCode=0 Jul 06 18:59:29.579935 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:29.579847 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlq9c" event={"ID":"27f875a5-8c79-42e8-b05f-819fe5dcc452","Type":"ContainerDied","Data":"4bd37f76e39e53c6bb8c7e60f84f8c5e6673951c72cb7d8ddd24cddd45491d84"} Jul 06 18:59:30.276822 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:30.276800 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:30.276950 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:30.276907 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hklfk" podUID="aab0c2d2-909e-466d-bd99-c4f4b3830060" Jul 06 18:59:30.276950 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:30.276807 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:30.277081 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:30.277060 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5xs7" podUID="f7d2aeda-aa6f-4865-ade0-f9aa24ea785a" Jul 06 18:59:30.583858 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:30.583823 2553 generic.go:356] "Generic (PLEG): container finished" podID="27f875a5-8c79-42e8-b05f-819fe5dcc452" containerID="eea4859d8a02791640bc685477e258fafaba5ee8c9a4aa63d3f9afe77e8821cf" exitCode=0 Jul 06 18:59:30.584276 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:30.583862 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlq9c" event={"ID":"27f875a5-8c79-42e8-b05f-819fe5dcc452","Type":"ContainerDied","Data":"eea4859d8a02791640bc685477e258fafaba5ee8c9a4aa63d3f9afe77e8821cf"} Jul 06 18:59:31.276738 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:31.276551 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:31.276907 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:31.276849 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmvjq" podUID="e7b6d1cb-2542-4861-9475-25365c8a5d15" Jul 06 18:59:31.588957 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:31.588885 2553 generic.go:356] "Generic (PLEG): container finished" podID="27f875a5-8c79-42e8-b05f-819fe5dcc452" containerID="1368bbb659d7b85bad3e47a5419c33149c0408b876c452dbf76e5e56e952d68b" exitCode=0 Jul 06 18:59:31.588957 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:31.588935 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlq9c" event={"ID":"27f875a5-8c79-42e8-b05f-819fe5dcc452","Type":"ContainerDied","Data":"1368bbb659d7b85bad3e47a5419c33149c0408b876c452dbf76e5e56e952d68b"} Jul 06 18:59:31.880571 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:31.880408 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret\") pod \"global-pull-secret-syncer-v5xs7\" (UID: \"f7d2aeda-aa6f-4865-ade0-f9aa24ea785a\") " pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:31.880571 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:31.880520 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs\") pod \"network-metrics-daemon-jmvjq\" (UID: \"e7b6d1cb-2542-4861-9475-25365c8a5d15\") " pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:31.880571 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:31.880556 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 18:59:31.880788 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:31.880602 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 18:59:31.880788 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:31.880629 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret podName:f7d2aeda-aa6f-4865-ade0-f9aa24ea785a nodeName:}" failed. No retries permitted until 2026-07-06 19:00:03.880610778 +0000 UTC m=+65.149793101 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret") pod "global-pull-secret-syncer-v5xs7" (UID: "f7d2aeda-aa6f-4865-ade0-f9aa24ea785a") : object "kube-system"/"original-pull-secret" not registered Jul 06 18:59:31.880788 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:31.880645 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs podName:e7b6d1cb-2542-4861-9475-25365c8a5d15 nodeName:}" failed. No retries permitted until 2026-07-06 19:00:03.880637255 +0000 UTC m=+65.149819574 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs") pod "network-metrics-daemon-jmvjq" (UID: "e7b6d1cb-2542-4861-9475-25365c8a5d15") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 18:59:31.981510 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:31.981478 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmgrx\" (UniqueName: \"kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx\") pod \"network-check-target-hklfk\" (UID: \"aab0c2d2-909e-466d-bd99-c4f4b3830060\") " pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:31.981660 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:31.981609 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 18:59:31.981660 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:31.981627 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 18:59:31.981660 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:31.981638 2553 projected.go:196] Error preparing data for projected volume kube-api-access-cmgrx for pod openshift-network-diagnostics/network-check-target-hklfk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 18:59:31.981750 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:31.981688 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx podName:aab0c2d2-909e-466d-bd99-c4f4b3830060 nodeName:}" failed. No retries permitted until 2026-07-06 19:00:03.981675877 +0000 UTC m=+65.250858197 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cmgrx" (UniqueName: "kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx") pod "network-check-target-hklfk" (UID: "aab0c2d2-909e-466d-bd99-c4f4b3830060") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 18:59:32.277336 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:32.277295 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:32.277537 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:32.277297 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:32.277537 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:32.277451 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hklfk" podUID="aab0c2d2-909e-466d-bd99-c4f4b3830060" Jul 06 18:59:32.277537 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:32.277497 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v5xs7" podUID="f7d2aeda-aa6f-4865-ade0-f9aa24ea785a" Jul 06 18:59:33.280108 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.280081 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:33.280532 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:33.280210 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmvjq" podUID="e7b6d1cb-2542-4861-9475-25365c8a5d15" Jul 06 18:59:33.549557 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.549529 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-237.ec2.internal" event="NodeReady" Jul 06 18:59:33.549700 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.549653 2553 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 18:59:33.607323 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.607290 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57b6b7754b-gx4l2"] Jul 06 18:59:33.609223 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.609165 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.614045 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.614015 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 18:59:33.614194 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.614061 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 18:59:33.614194 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.614112 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 18:59:33.619093 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.619075 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ch228\"" Jul 06 18:59:33.623599 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.623575 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 18:59:33.635305 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.635274 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-tsj58"] Jul 06 18:59:33.637084 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.637052 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v"] Jul 06 18:59:33.637254 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.637174 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.638770 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.638749 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7456bbb6dc-zfg4t"] Jul 06 18:59:33.638893 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.638868 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" Jul 06 18:59:33.640309 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.640288 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b4bv7"] Jul 06 18:59:33.640444 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.640404 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:33.641888 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.641869 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-jjlwh"] Jul 06 18:59:33.642024 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.641994 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b4bv7" Jul 06 18:59:33.643234 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.643216 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn"] Jul 06 18:59:33.643364 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.643348 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-jjlwh" Jul 06 18:59:33.644706 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.644686 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57b6b7754b-gx4l2"] Jul 06 18:59:33.644832 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.644717 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-8m9wb"] Jul 06 18:59:33.644891 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.644847 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn" Jul 06 18:59:33.646375 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.646357 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" Jul 06 18:59:33.655312 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.655292 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 18:59:33.655521 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.655451 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 18:59:33.655521 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.655506 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 06 18:59:33.655681 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.655658 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 06 18:59:33.655681 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.655672 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 06 18:59:33.655776 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.655673 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 06 18:59:33.655776 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.655676 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 06 18:59:33.655776 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.655742 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 06 18:59:33.656052 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.656027 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 06 18:59:33.656139 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.656089 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 06 18:59:33.656139 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.656132 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 18:59:33.656240 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.656138 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 18:59:33.656240 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.656153 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 06 18:59:33.656537 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.656398 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-tz9gj\"" Jul 06 18:59:33.656537 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.656411 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 06 18:59:33.656537 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.656446 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 06 18:59:33.656537 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.656454 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 06 18:59:33.656867 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.656557 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 06 18:59:33.656867 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.656602 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xfsk6\"" Jul 06 18:59:33.656867 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.656666 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5xg49\"" Jul 06 18:59:33.656867 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.656691 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 06 18:59:33.656867 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.656728 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 06 18:59:33.656867 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.656764 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 06 18:59:33.656867 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.656862 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-z9hg5\"" Jul 06 18:59:33.657167 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.656902 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 06 18:59:33.657167 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.657046 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 06 18:59:33.657167 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.657098 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8l6kc\"" Jul 06 18:59:33.657167 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.657136 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 06 18:59:33.657316 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.657190 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 18:59:33.657496 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.657398 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 06 18:59:33.657496 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.657463 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hfrrk\"" Jul 06 18:59:33.657496 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.657403 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-dm6hl\"" Jul 06 18:59:33.657496 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.657466 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 06 18:59:33.657707 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.657472 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 18:59:33.666501 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.666478 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-8m9wb"] Jul 06 18:59:33.668005 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.667977 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 06 18:59:33.673049 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.672983 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 06 18:59:33.674196 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.674148 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b4bv7"] Jul 06 18:59:33.674950 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.674929 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn"] Jul 06 18:59:33.676326 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.676301 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-jjlwh"] Jul 06 18:59:33.685203 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.685179 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v"] Jul 06 18:59:33.692935 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.692913 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.693029 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.692943 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r92tv\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-kube-api-access-r92tv\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.693029 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.692967 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-image-registry-private-configuration\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.693029 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.693015 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-trusted-ca\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.693128 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.693060 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-certificates\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.693128 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.693094 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-bound-sa-token\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.693185 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.693140 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-ca-trust-extracted\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.693185 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.693161 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-installation-pull-secrets\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.702974 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.702951 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-tsj58"] Jul 06 18:59:33.704718 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.704698 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mkqz8"] Jul 06 18:59:33.706890 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.706871 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s8v9j"] Jul 06 18:59:33.706996 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.706981 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mkqz8" Jul 06 18:59:33.708439 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.708405 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-5t7tr"] Jul 06 18:59:33.708499 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.708489 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s8v9j" Jul 06 18:59:33.710219 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.710197 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 18:59:33.710296 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.710261 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sc9lq"] Jul 06 18:59:33.710491 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.710470 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2f7gj\"" Jul 06 18:59:33.710547 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.710507 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-5t7tr" Jul 06 18:59:33.712207 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.712189 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7456bbb6dc-zfg4t"] Jul 06 18:59:33.712288 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.712276 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sc9lq" Jul 06 18:59:33.712847 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.712830 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 18:59:33.716551 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.716533 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 06 18:59:33.716660 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.716558 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 06 18:59:33.716660 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.716574 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8hlcx\"" Jul 06 18:59:33.717038 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.717020 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 06 18:59:33.717137 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.717040 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 18:59:33.717137 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.717047 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-r74wt\"" Jul 06 18:59:33.717137 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.717045 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 18:59:33.717378 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.717362 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 18:59:33.718138 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.718124 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-r2qkq\"" Jul 06 18:59:33.720086 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.719923 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 06 18:59:33.720086 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.719931 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 18:59:33.720721 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.720701 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-5t7tr"] Jul 06 18:59:33.723526 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.723510 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 18:59:33.727209 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.727189 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mkqz8"] Jul 06 18:59:33.737933 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.737913 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sc9lq"] Jul 06 18:59:33.759935 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.759906 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s8v9j"] Jul 06 18:59:33.793613 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.793574 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r92tv\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-kube-api-access-r92tv\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.793613 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.793616 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-stats-auth\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:33.793832 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.793644 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/628f818d-2286-4af1-b946-5779244f0bff-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-s8v9j\" (UID: \"628f818d-2286-4af1-b946-5779244f0bff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s8v9j" Jul 06 18:59:33.793832 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.793698 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d11343a5-a7ee-47e9-8875-d6e7ae822930-cert\") pod \"ingress-canary-sc9lq\" (UID: \"d11343a5-a7ee-47e9-8875-d6e7ae822930\") " pod="openshift-ingress-canary/ingress-canary-sc9lq" Jul 06 18:59:33.793832 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.793739 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nswwg\" (UniqueName: \"kubernetes.io/projected/89edb5ad-301f-4971-a47a-82f8078b0c21-kube-api-access-nswwg\") pod \"cluster-samples-operator-56fcf86bf-rzpjn\" (UID: \"89edb5ad-301f-4971-a47a-82f8078b0c21\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn" Jul 06 18:59:33.793832 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.793772 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0c10df95-ff62-4f94-a74d-91742dc4c92c-config\") pod \"service-ca-operator-5d46d9ffd4-jjlwh\" (UID: \"0c10df95-ff62-4f94-a74d-91742dc4c92c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-jjlwh" Jul 06 18:59:33.793832 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.793814 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-bound-sa-token\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.794060 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.793839 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-s259v\" (UID: \"2caf149a-9d45-4412-8c6f-006d28f81ef1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" Jul 06 18:59:33.794060 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.793887 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-rzpjn\" (UID: \"89edb5ad-301f-4971-a47a-82f8078b0c21\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn" Jul 06 18:59:33.794060 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.793908 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/724d10df-ec14-4dcd-8b9c-5265acf5ed80-tmp\") pod \"insights-operator-5c847dcfc4-tsj58\" (UID: \"724d10df-ec14-4dcd-8b9c-5265acf5ed80\") " pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.794060 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.793943 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k67q2\" (UniqueName: \"kubernetes.io/projected/628f818d-2286-4af1-b946-5779244f0bff-kube-api-access-k67q2\") pod \"kube-storage-version-migrator-operator-846bf668f7-s8v9j\" (UID: \"628f818d-2286-4af1-b946-5779244f0bff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s8v9j" Jul 06 18:59:33.794060 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.793962 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd4qj\" (UniqueName: \"kubernetes.io/projected/d11343a5-a7ee-47e9-8875-d6e7ae822930-kube-api-access-pd4qj\") pod \"ingress-canary-sc9lq\" (UID: \"d11343a5-a7ee-47e9-8875-d6e7ae822930\") " pod="openshift-ingress-canary/ingress-canary-sc9lq" Jul 06 18:59:33.794060 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.794027 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.794281 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.794065 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:33.794281 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.794085 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0c10df95-ff62-4f94-a74d-91742dc4c92c-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-jjlwh\" (UID: \"0c10df95-ff62-4f94-a74d-91742dc4c92c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-jjlwh" Jul 06 18:59:33.794281 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.794102 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5dxl\" (UniqueName: \"kubernetes.io/projected/41eea561-3913-46d4-beca-714c42bc07cd-kube-api-access-g5dxl\") pod \"dns-default-mkqz8\" (UID: \"41eea561-3913-46d4-beca-714c42bc07cd\") " pod="openshift-dns/dns-default-mkqz8" Jul 06 18:59:33.794281 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.794141 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbwf2\" (UniqueName: \"kubernetes.io/projected/1962c753-8b1d-4ac7-9e03-9756f80ef241-kube-api-access-lbwf2\") pod \"network-check-source-989f8b8dc-5t7tr\" (UID: \"1962c753-8b1d-4ac7-9e03-9756f80ef241\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-5t7tr" Jul 06 18:59:33.794281 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.794174 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/724d10df-ec14-4dcd-8b9c-5265acf5ed80-snapshots\") pod \"insights-operator-5c847dcfc4-tsj58\" (UID: \"724d10df-ec14-4dcd-8b9c-5265acf5ed80\") " pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.794281 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.794232 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:33.794281 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.794274 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/724d10df-ec14-4dcd-8b9c-5265acf5ed80-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-tsj58\" (UID: \"724d10df-ec14-4dcd-8b9c-5265acf5ed80\") " pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.794548 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.794305 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/724d10df-ec14-4dcd-8b9c-5265acf5ed80-serving-cert\") pod \"insights-operator-5c847dcfc4-tsj58\" (UID: \"724d10df-ec14-4dcd-8b9c-5265acf5ed80\") " pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.794548 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.794327 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2caf149a-9d45-4412-8c6f-006d28f81ef1-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-s259v\" (UID: \"2caf149a-9d45-4412-8c6f-006d28f81ef1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" Jul 06 18:59:33.794548 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.794343 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-default-certificate\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:33.794548 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:33.794351 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 18:59:33.794548 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:33.794362 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b6b7754b-gx4l2: secret "image-registry-tls" not found Jul 06 18:59:33.794548 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.794365 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1923cd02-24e0-4753-be72-cb22fbd5c05f-config\") pod \"console-operator-68b757fb67-8m9wb\" (UID: \"1923cd02-24e0-4753-be72-cb22fbd5c05f\") " pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" Jul 06 18:59:33.794548 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:33.794450 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls podName:5c9d5b1a-9817-4d71-a7b4-ff6ced34939c nodeName:}" failed. No retries permitted until 2026-07-06 18:59:34.294408002 +0000 UTC m=+35.563590339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls") pod "image-registry-57b6b7754b-gx4l2" (UID: "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c") : secret "image-registry-tls" not found Jul 06 18:59:33.794548 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.794491 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/724d10df-ec14-4dcd-8b9c-5265acf5ed80-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-tsj58\" (UID: \"724d10df-ec14-4dcd-8b9c-5265acf5ed80\") " pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.794548 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.794523 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-image-registry-private-configuration\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.794785 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.794551 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-trusted-ca\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.795056 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.794656 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjstc\" (UniqueName: \"kubernetes.io/projected/0c10df95-ff62-4f94-a74d-91742dc4c92c-kube-api-access-bjstc\") pod \"service-ca-operator-5d46d9ffd4-jjlwh\" (UID: \"0c10df95-ff62-4f94-a74d-91742dc4c92c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-jjlwh" Jul 06 18:59:33.795056 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.795014 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-certificates\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.795056 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.795056 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htqjb\" (UniqueName: \"kubernetes.io/projected/13153a33-8136-473a-b96d-7bd6b1e8c5e3-kube-api-access-htqjb\") pod \"volume-data-source-validator-5d456fdbcd-b4bv7\" (UID: \"13153a33-8136-473a-b96d-7bd6b1e8c5e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b4bv7" Jul 06 18:59:33.795277 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.795089 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls\") pod \"dns-default-mkqz8\" (UID: \"41eea561-3913-46d4-beca-714c42bc07cd\") " pod="openshift-dns/dns-default-mkqz8" Jul 06 18:59:33.795277 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.795119 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1923cd02-24e0-4753-be72-cb22fbd5c05f-serving-cert\") pod \"console-operator-68b757fb67-8m9wb\" (UID: \"1923cd02-24e0-4753-be72-cb22fbd5c05f\") " pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" Jul 06 18:59:33.795277 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.795146 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cmqg\" (UniqueName: \"kubernetes.io/projected/1923cd02-24e0-4753-be72-cb22fbd5c05f-kube-api-access-4cmqg\") pod \"console-operator-68b757fb67-8m9wb\" (UID: \"1923cd02-24e0-4753-be72-cb22fbd5c05f\") " pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" Jul 06 18:59:33.795277 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.795178 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41eea561-3913-46d4-beca-714c42bc07cd-config-volume\") pod \"dns-default-mkqz8\" (UID: \"41eea561-3913-46d4-beca-714c42bc07cd\") " pod="openshift-dns/dns-default-mkqz8" Jul 06 18:59:33.795277 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.795211 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdkpb\" (UniqueName: \"kubernetes.io/projected/b78c33c0-d4f4-4a97-8621-90904852db08-kube-api-access-cdkpb\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:33.795529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.795290 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1923cd02-24e0-4753-be72-cb22fbd5c05f-trusted-ca\") pod \"console-operator-68b757fb67-8m9wb\" (UID: \"1923cd02-24e0-4753-be72-cb22fbd5c05f\") " pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" Jul 06 18:59:33.795529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.795330 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-ca-trust-extracted\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.795529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.795368 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmtpq\" (UniqueName: \"kubernetes.io/projected/2caf149a-9d45-4412-8c6f-006d28f81ef1-kube-api-access-hmtpq\") pod \"cluster-monitoring-operator-64786d9cbb-s259v\" (UID: \"2caf149a-9d45-4412-8c6f-006d28f81ef1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" Jul 06 18:59:33.795529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.795405 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41eea561-3913-46d4-beca-714c42bc07cd-tmp-dir\") pod \"dns-default-mkqz8\" (UID: \"41eea561-3913-46d4-beca-714c42bc07cd\") " pod="openshift-dns/dns-default-mkqz8" Jul 06 18:59:33.795529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.795454 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9r6s\" (UniqueName: \"kubernetes.io/projected/724d10df-ec14-4dcd-8b9c-5265acf5ed80-kube-api-access-c9r6s\") pod \"insights-operator-5c847dcfc4-tsj58\" (UID: \"724d10df-ec14-4dcd-8b9c-5265acf5ed80\") " pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.795529 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.795491 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-installation-pull-secrets\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.795779 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.795536 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/628f818d-2286-4af1-b946-5779244f0bff-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-s8v9j\" (UID: \"628f818d-2286-4af1-b946-5779244f0bff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s8v9j" Jul 06 18:59:33.795834 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.795820 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-certificates\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.795969 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.795922 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-ca-trust-extracted\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.796150 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.796112 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-trusted-ca\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.800339 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.800289 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-installation-pull-secrets\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.800339 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.800294 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-image-registry-private-configuration\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.813471 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.813438 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-bound-sa-token\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.813662 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.813544 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r92tv\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-kube-api-access-r92tv\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:33.896548 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.896507 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k67q2\" (UniqueName: \"kubernetes.io/projected/628f818d-2286-4af1-b946-5779244f0bff-kube-api-access-k67q2\") pod \"kube-storage-version-migrator-operator-846bf668f7-s8v9j\" (UID: \"628f818d-2286-4af1-b946-5779244f0bff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s8v9j" Jul 06 18:59:33.896548 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.896544 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd4qj\" (UniqueName: \"kubernetes.io/projected/d11343a5-a7ee-47e9-8875-d6e7ae822930-kube-api-access-pd4qj\") pod \"ingress-canary-sc9lq\" (UID: \"d11343a5-a7ee-47e9-8875-d6e7ae822930\") " pod="openshift-ingress-canary/ingress-canary-sc9lq" Jul 06 18:59:33.896769 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.896593 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:33.896769 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.896618 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0c10df95-ff62-4f94-a74d-91742dc4c92c-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-jjlwh\" (UID: \"0c10df95-ff62-4f94-a74d-91742dc4c92c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-jjlwh" Jul 06 18:59:33.896769 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.896644 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5dxl\" (UniqueName: \"kubernetes.io/projected/41eea561-3913-46d4-beca-714c42bc07cd-kube-api-access-g5dxl\") pod \"dns-default-mkqz8\" (UID: \"41eea561-3913-46d4-beca-714c42bc07cd\") " pod="openshift-dns/dns-default-mkqz8" Jul 06 18:59:33.896769 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.896668 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbwf2\" (UniqueName: \"kubernetes.io/projected/1962c753-8b1d-4ac7-9e03-9756f80ef241-kube-api-access-lbwf2\") pod \"network-check-source-989f8b8dc-5t7tr\" (UID: \"1962c753-8b1d-4ac7-9e03-9756f80ef241\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-5t7tr" Jul 06 18:59:33.896769 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.896692 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/724d10df-ec14-4dcd-8b9c-5265acf5ed80-snapshots\") pod \"insights-operator-5c847dcfc4-tsj58\" (UID: \"724d10df-ec14-4dcd-8b9c-5265acf5ed80\") " pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.896769 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:33.896709 2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 18:59:33.896769 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.896727 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:33.896769 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.896751 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/724d10df-ec14-4dcd-8b9c-5265acf5ed80-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-tsj58\" (UID: \"724d10df-ec14-4dcd-8b9c-5265acf5ed80\") " pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.897167 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:33.896788 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs podName:b78c33c0-d4f4-4a97-8621-90904852db08 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:34.396767966 +0000 UTC m=+35.665950311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs") pod "router-default-7456bbb6dc-zfg4t" (UID: "b78c33c0-d4f4-4a97-8621-90904852db08") : secret "router-metrics-certs-default" not found Jul 06 18:59:33.897167 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.896809 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/724d10df-ec14-4dcd-8b9c-5265acf5ed80-serving-cert\") pod \"insights-operator-5c847dcfc4-tsj58\" (UID: \"724d10df-ec14-4dcd-8b9c-5265acf5ed80\") " pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.897167 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.896841 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2caf149a-9d45-4412-8c6f-006d28f81ef1-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-s259v\" (UID: \"2caf149a-9d45-4412-8c6f-006d28f81ef1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" Jul 06 18:59:33.897167 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:33.896883 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle podName:b78c33c0-d4f4-4a97-8621-90904852db08 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:34.396863043 +0000 UTC m=+35.666045389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle") pod "router-default-7456bbb6dc-zfg4t" (UID: "b78c33c0-d4f4-4a97-8621-90904852db08") : configmap references non-existent config key: service-ca.crt Jul 06 18:59:33.897167 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.896914 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-default-certificate\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:33.897167 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.896946 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1923cd02-24e0-4753-be72-cb22fbd5c05f-config\") pod \"console-operator-68b757fb67-8m9wb\" (UID: \"1923cd02-24e0-4753-be72-cb22fbd5c05f\") " pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" Jul 06 18:59:33.897167 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.896978 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/724d10df-ec14-4dcd-8b9c-5265acf5ed80-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-tsj58\" (UID: \"724d10df-ec14-4dcd-8b9c-5265acf5ed80\") " pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.897167 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897103 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjstc\" (UniqueName: \"kubernetes.io/projected/0c10df95-ff62-4f94-a74d-91742dc4c92c-kube-api-access-bjstc\") pod \"service-ca-operator-5d46d9ffd4-jjlwh\" (UID: \"0c10df95-ff62-4f94-a74d-91742dc4c92c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-jjlwh" Jul 06 18:59:33.897167 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897141 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htqjb\" (UniqueName: \"kubernetes.io/projected/13153a33-8136-473a-b96d-7bd6b1e8c5e3-kube-api-access-htqjb\") pod \"volume-data-source-validator-5d456fdbcd-b4bv7\" (UID: \"13153a33-8136-473a-b96d-7bd6b1e8c5e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b4bv7" Jul 06 18:59:33.897167 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897165 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls\") pod \"dns-default-mkqz8\" (UID: \"41eea561-3913-46d4-beca-714c42bc07cd\") " pod="openshift-dns/dns-default-mkqz8" Jul 06 18:59:33.897673 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897191 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1923cd02-24e0-4753-be72-cb22fbd5c05f-serving-cert\") pod \"console-operator-68b757fb67-8m9wb\" (UID: \"1923cd02-24e0-4753-be72-cb22fbd5c05f\") " pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" Jul 06 18:59:33.897673 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897221 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cmqg\" (UniqueName: \"kubernetes.io/projected/1923cd02-24e0-4753-be72-cb22fbd5c05f-kube-api-access-4cmqg\") pod \"console-operator-68b757fb67-8m9wb\" (UID: \"1923cd02-24e0-4753-be72-cb22fbd5c05f\") " pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" Jul 06 18:59:33.897673 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897248 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41eea561-3913-46d4-beca-714c42bc07cd-config-volume\") pod \"dns-default-mkqz8\" (UID: \"41eea561-3913-46d4-beca-714c42bc07cd\") " pod="openshift-dns/dns-default-mkqz8" Jul 06 18:59:33.897673 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897277 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdkpb\" (UniqueName: \"kubernetes.io/projected/b78c33c0-d4f4-4a97-8621-90904852db08-kube-api-access-cdkpb\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:33.897673 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897304 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1923cd02-24e0-4753-be72-cb22fbd5c05f-trusted-ca\") pod \"console-operator-68b757fb67-8m9wb\" (UID: \"1923cd02-24e0-4753-be72-cb22fbd5c05f\") " pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" Jul 06 18:59:33.897673 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897335 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmtpq\" (UniqueName: \"kubernetes.io/projected/2caf149a-9d45-4412-8c6f-006d28f81ef1-kube-api-access-hmtpq\") pod \"cluster-monitoring-operator-64786d9cbb-s259v\" (UID: \"2caf149a-9d45-4412-8c6f-006d28f81ef1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" Jul 06 18:59:33.897673 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897358 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41eea561-3913-46d4-beca-714c42bc07cd-tmp-dir\") pod \"dns-default-mkqz8\" (UID: \"41eea561-3913-46d4-beca-714c42bc07cd\") " pod="openshift-dns/dns-default-mkqz8" Jul 06 18:59:33.897673 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897381 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9r6s\" (UniqueName: \"kubernetes.io/projected/724d10df-ec14-4dcd-8b9c-5265acf5ed80-kube-api-access-c9r6s\") pod \"insights-operator-5c847dcfc4-tsj58\" (UID: \"724d10df-ec14-4dcd-8b9c-5265acf5ed80\") " pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.897673 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897440 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/628f818d-2286-4af1-b946-5779244f0bff-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-s8v9j\" (UID: \"628f818d-2286-4af1-b946-5779244f0bff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s8v9j" Jul 06 18:59:33.897673 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897476 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-stats-auth\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:33.897673 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897489 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/724d10df-ec14-4dcd-8b9c-5265acf5ed80-snapshots\") pod \"insights-operator-5c847dcfc4-tsj58\" (UID: \"724d10df-ec14-4dcd-8b9c-5265acf5ed80\") " pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.897673 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897501 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/628f818d-2286-4af1-b946-5779244f0bff-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-s8v9j\" (UID: \"628f818d-2286-4af1-b946-5779244f0bff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s8v9j" Jul 06 18:59:33.897673 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897528 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d11343a5-a7ee-47e9-8875-d6e7ae822930-cert\") pod \"ingress-canary-sc9lq\" (UID: \"d11343a5-a7ee-47e9-8875-d6e7ae822930\") " pod="openshift-ingress-canary/ingress-canary-sc9lq" Jul 06 18:59:33.897673 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897559 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nswwg\" (UniqueName: \"kubernetes.io/projected/89edb5ad-301f-4971-a47a-82f8078b0c21-kube-api-access-nswwg\") pod \"cluster-samples-operator-56fcf86bf-rzpjn\" (UID: \"89edb5ad-301f-4971-a47a-82f8078b0c21\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn" Jul 06 18:59:33.897673 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:33.897581 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 18:59:33.897673 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:33.897624 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls podName:41eea561-3913-46d4-beca-714c42bc07cd nodeName:}" failed. No retries permitted until 2026-07-06 18:59:34.397614301 +0000 UTC m=+35.666796649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls") pod "dns-default-mkqz8" (UID: "41eea561-3913-46d4-beca-714c42bc07cd") : secret "dns-default-metrics-tls" not found Jul 06 18:59:33.898457 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897640 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/724d10df-ec14-4dcd-8b9c-5265acf5ed80-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-tsj58\" (UID: \"724d10df-ec14-4dcd-8b9c-5265acf5ed80\") " pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.898457 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897826 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2caf149a-9d45-4412-8c6f-006d28f81ef1-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-s259v\" (UID: \"2caf149a-9d45-4412-8c6f-006d28f81ef1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" Jul 06 18:59:33.898457 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897581 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0c10df95-ff62-4f94-a74d-91742dc4c92c-config\") pod \"service-ca-operator-5d46d9ffd4-jjlwh\" (UID: \"0c10df95-ff62-4f94-a74d-91742dc4c92c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-jjlwh" Jul 06 18:59:33.898457 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897897 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-s259v\" (UID: \"2caf149a-9d45-4412-8c6f-006d28f81ef1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" Jul 06 18:59:33.898457 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897921 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-rzpjn\" (UID: \"89edb5ad-301f-4971-a47a-82f8078b0c21\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn" Jul 06 18:59:33.898457 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.897941 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/724d10df-ec14-4dcd-8b9c-5265acf5ed80-tmp\") pod \"insights-operator-5c847dcfc4-tsj58\" (UID: \"724d10df-ec14-4dcd-8b9c-5265acf5ed80\") " pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.898457 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.898017 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41eea561-3913-46d4-beca-714c42bc07cd-config-volume\") pod \"dns-default-mkqz8\" (UID: \"41eea561-3913-46d4-beca-714c42bc07cd\") " pod="openshift-dns/dns-default-mkqz8" Jul 06 18:59:33.898457 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.898081 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/724d10df-ec14-4dcd-8b9c-5265acf5ed80-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-tsj58\" (UID: \"724d10df-ec14-4dcd-8b9c-5265acf5ed80\") " pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.898457 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.898127 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0c10df95-ff62-4f94-a74d-91742dc4c92c-config\") pod \"service-ca-operator-5d46d9ffd4-jjlwh\" (UID: \"0c10df95-ff62-4f94-a74d-91742dc4c92c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-jjlwh" Jul 06 18:59:33.898457 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.898430 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/724d10df-ec14-4dcd-8b9c-5265acf5ed80-tmp\") pod \"insights-operator-5c847dcfc4-tsj58\" (UID: \"724d10df-ec14-4dcd-8b9c-5265acf5ed80\") " pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.898924 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:33.898522 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 18:59:33.898924 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:33.898564 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls podName:2caf149a-9d45-4412-8c6f-006d28f81ef1 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:34.398551242 +0000 UTC m=+35.667733580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-s259v" (UID: "2caf149a-9d45-4412-8c6f-006d28f81ef1") : secret "cluster-monitoring-operator-tls" not found Jul 06 18:59:33.898924 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:33.898630 2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 18:59:33.898924 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:33.898665 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls podName:89edb5ad-301f-4971-a47a-82f8078b0c21 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:34.398653651 +0000 UTC m=+35.667836003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-rzpjn" (UID: "89edb5ad-301f-4971-a47a-82f8078b0c21") : secret "samples-operator-tls" not found Jul 06 18:59:33.899083 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.899055 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1923cd02-24e0-4753-be72-cb22fbd5c05f-config\") pod \"console-operator-68b757fb67-8m9wb\" (UID: \"1923cd02-24e0-4753-be72-cb22fbd5c05f\") " pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" Jul 06 18:59:33.899255 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.899231 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/628f818d-2286-4af1-b946-5779244f0bff-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-s8v9j\" (UID: \"628f818d-2286-4af1-b946-5779244f0bff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s8v9j" Jul 06 18:59:33.900071 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.899854 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0c10df95-ff62-4f94-a74d-91742dc4c92c-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-jjlwh\" (UID: \"0c10df95-ff62-4f94-a74d-91742dc4c92c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-jjlwh" Jul 06 18:59:33.900071 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.900007 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1923cd02-24e0-4753-be72-cb22fbd5c05f-trusted-ca\") pod \"console-operator-68b757fb67-8m9wb\" (UID: \"1923cd02-24e0-4753-be72-cb22fbd5c05f\") " pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" Jul 06 18:59:33.900646 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.900478 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41eea561-3913-46d4-beca-714c42bc07cd-tmp-dir\") pod \"dns-default-mkqz8\" (UID: \"41eea561-3913-46d4-beca-714c42bc07cd\") " pod="openshift-dns/dns-default-mkqz8" Jul 06 18:59:33.901865 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.901822 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-default-certificate\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:33.901955 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.901887 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/724d10df-ec14-4dcd-8b9c-5265acf5ed80-serving-cert\") pod \"insights-operator-5c847dcfc4-tsj58\" (UID: \"724d10df-ec14-4dcd-8b9c-5265acf5ed80\") " pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.902128 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.902104 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/628f818d-2286-4af1-b946-5779244f0bff-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-s8v9j\" (UID: \"628f818d-2286-4af1-b946-5779244f0bff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s8v9j" Jul 06 18:59:33.902501 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.902481 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d11343a5-a7ee-47e9-8875-d6e7ae822930-cert\") pod \"ingress-canary-sc9lq\" (UID: \"d11343a5-a7ee-47e9-8875-d6e7ae822930\") " pod="openshift-ingress-canary/ingress-canary-sc9lq" Jul 06 18:59:33.903021 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.903004 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-stats-auth\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:33.903329 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.903314 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1923cd02-24e0-4753-be72-cb22fbd5c05f-serving-cert\") pod \"console-operator-68b757fb67-8m9wb\" (UID: \"1923cd02-24e0-4753-be72-cb22fbd5c05f\") " pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" Jul 06 18:59:33.920777 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.920713 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nswwg\" (UniqueName: \"kubernetes.io/projected/89edb5ad-301f-4971-a47a-82f8078b0c21-kube-api-access-nswwg\") pod \"cluster-samples-operator-56fcf86bf-rzpjn\" (UID: \"89edb5ad-301f-4971-a47a-82f8078b0c21\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn" Jul 06 18:59:33.923919 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.923893 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9r6s\" (UniqueName: \"kubernetes.io/projected/724d10df-ec14-4dcd-8b9c-5265acf5ed80-kube-api-access-c9r6s\") pod \"insights-operator-5c847dcfc4-tsj58\" (UID: \"724d10df-ec14-4dcd-8b9c-5265acf5ed80\") " pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.924069 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.924037 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htqjb\" (UniqueName: \"kubernetes.io/projected/13153a33-8136-473a-b96d-7bd6b1e8c5e3-kube-api-access-htqjb\") pod \"volume-data-source-validator-5d456fdbcd-b4bv7\" (UID: \"13153a33-8136-473a-b96d-7bd6b1e8c5e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b4bv7" Jul 06 18:59:33.924380 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.924352 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbwf2\" (UniqueName: \"kubernetes.io/projected/1962c753-8b1d-4ac7-9e03-9756f80ef241-kube-api-access-lbwf2\") pod \"network-check-source-989f8b8dc-5t7tr\" (UID: \"1962c753-8b1d-4ac7-9e03-9756f80ef241\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-5t7tr" Jul 06 18:59:33.924621 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.924601 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdkpb\" (UniqueName: \"kubernetes.io/projected/b78c33c0-d4f4-4a97-8621-90904852db08-kube-api-access-cdkpb\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:33.926228 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.926181 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k67q2\" (UniqueName: \"kubernetes.io/projected/628f818d-2286-4af1-b946-5779244f0bff-kube-api-access-k67q2\") pod \"kube-storage-version-migrator-operator-846bf668f7-s8v9j\" (UID: \"628f818d-2286-4af1-b946-5779244f0bff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s8v9j" Jul 06 18:59:33.926955 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.926934 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmtpq\" (UniqueName: \"kubernetes.io/projected/2caf149a-9d45-4412-8c6f-006d28f81ef1-kube-api-access-hmtpq\") pod \"cluster-monitoring-operator-64786d9cbb-s259v\" (UID: \"2caf149a-9d45-4412-8c6f-006d28f81ef1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" Jul 06 18:59:33.927386 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.927361 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cmqg\" (UniqueName: \"kubernetes.io/projected/1923cd02-24e0-4753-be72-cb22fbd5c05f-kube-api-access-4cmqg\") pod \"console-operator-68b757fb67-8m9wb\" (UID: \"1923cd02-24e0-4753-be72-cb22fbd5c05f\") " pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" Jul 06 18:59:33.927632 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.927610 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd4qj\" (UniqueName: \"kubernetes.io/projected/d11343a5-a7ee-47e9-8875-d6e7ae822930-kube-api-access-pd4qj\") pod \"ingress-canary-sc9lq\" (UID: \"d11343a5-a7ee-47e9-8875-d6e7ae822930\") " pod="openshift-ingress-canary/ingress-canary-sc9lq" Jul 06 18:59:33.927731 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.927717 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5dxl\" (UniqueName: \"kubernetes.io/projected/41eea561-3913-46d4-beca-714c42bc07cd-kube-api-access-g5dxl\") pod \"dns-default-mkqz8\" (UID: \"41eea561-3913-46d4-beca-714c42bc07cd\") " pod="openshift-dns/dns-default-mkqz8" Jul 06 18:59:33.927982 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.927951 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjstc\" (UniqueName: \"kubernetes.io/projected/0c10df95-ff62-4f94-a74d-91742dc4c92c-kube-api-access-bjstc\") pod \"service-ca-operator-5d46d9ffd4-jjlwh\" (UID: \"0c10df95-ff62-4f94-a74d-91742dc4c92c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-jjlwh" Jul 06 18:59:33.947645 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.947621 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" Jul 06 18:59:33.959637 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.959607 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gm25_99098e94-f6ae-4918-b007-e85999dfe259/konnectivity-agent/0.log" limit=1024 Jul 06 18:59:33.968990 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.968969 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b4bv7" Jul 06 18:59:33.976745 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.976717 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-jjlwh" Jul 06 18:59:33.989277 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:33.989247 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" Jul 06 18:59:34.024380 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.024158 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s8v9j" Jul 06 18:59:34.034574 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.030904 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-5t7tr" Jul 06 18:59:34.035633 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.035390 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sc9lq" Jul 06 18:59:34.161018 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.160980 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gm25_99098e94-f6ae-4918-b007-e85999dfe259/konnectivity-agent/0.log" limit=1024 Jul 06 18:59:34.171075 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.170614 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-tsj58"] Jul 06 18:59:34.184307 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:59:34.184010 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724d10df_ec14_4dcd_8b9c_5265acf5ed80.slice/crio-d83541a364be4022d878ef2783bc708c92192a470663a974195f39020b301e18 WatchSource:0}: Error finding container d83541a364be4022d878ef2783bc708c92192a470663a974195f39020b301e18: Status 404 returned error can't find the container with id d83541a364be4022d878ef2783bc708c92192a470663a974195f39020b301e18 Jul 06 18:59:34.199793 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.199721 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b4bv7"] Jul 06 18:59:34.201331 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.201308 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-jjlwh"] Jul 06 18:59:34.254152 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.254121 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-8m9wb"] Jul 06 18:59:34.261733 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:59:34.261661 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1923cd02_24e0_4753_be72_cb22fbd5c05f.slice/crio-021ec36f1a2b086df4f84c307ad347321326b8acf0353190ba0ba0bb0b64a490 WatchSource:0}: Error finding container 021ec36f1a2b086df4f84c307ad347321326b8acf0353190ba0ba0bb0b64a490: Status 404 returned error can't find the container with id 021ec36f1a2b086df4f84c307ad347321326b8acf0353190ba0ba0bb0b64a490 Jul 06 18:59:34.277898 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.277857 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 18:59:34.278233 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.278219 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 18:59:34.282578 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.282211 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 18:59:34.282578 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.282301 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zn9xt\"" Jul 06 18:59:34.303408 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.303378 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:34.303565 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:34.303516 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 18:59:34.303565 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:34.303534 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b6b7754b-gx4l2: secret "image-registry-tls" not found Jul 06 18:59:34.303920 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:34.303884 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls podName:5c9d5b1a-9817-4d71-a7b4-ff6ced34939c nodeName:}" failed. No retries permitted until 2026-07-06 18:59:35.303853789 +0000 UTC m=+36.573036127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls") pod "image-registry-57b6b7754b-gx4l2" (UID: "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c") : secret "image-registry-tls" not found Jul 06 18:59:34.309148 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.309121 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-5t7tr"] Jul 06 18:59:34.312974 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:59:34.312952 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1962c753_8b1d_4ac7_9e03_9756f80ef241.slice/crio-6be391f5faeedbd871595668f7230faadd86b1b9b0305dddd067ba082cd3e33d WatchSource:0}: Error finding container 6be391f5faeedbd871595668f7230faadd86b1b9b0305dddd067ba082cd3e33d: Status 404 returned error can't find the container with id 6be391f5faeedbd871595668f7230faadd86b1b9b0305dddd067ba082cd3e33d Jul 06 18:59:34.318016 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.317982 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s8v9j"] Jul 06 18:59:34.321197 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:59:34.321176 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628f818d_2286_4af1_b946_5779244f0bff.slice/crio-0723952afab54bafed618c730ad62c5a929e0d421750fc69bdf4ddb46155ae65 WatchSource:0}: Error finding container 0723952afab54bafed618c730ad62c5a929e0d421750fc69bdf4ddb46155ae65: Status 404 returned error can't find the container with id 0723952afab54bafed618c730ad62c5a929e0d421750fc69bdf4ddb46155ae65 Jul 06 18:59:34.327575 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.327550 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sc9lq"] Jul 06 18:59:34.331049 ip-10-0-141-237 kubenswrapper[2553]: W0706 18:59:34.331025 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11343a5_a7ee_47e9_8875_d6e7ae822930.slice/crio-7dd9be617ef6e3eb2d62af6916fdb16840dd544f4e17ff28d790f550ced29016 WatchSource:0}: Error finding container 7dd9be617ef6e3eb2d62af6916fdb16840dd544f4e17ff28d790f550ced29016: Status 404 returned error can't find the container with id 7dd9be617ef6e3eb2d62af6916fdb16840dd544f4e17ff28d790f550ced29016 Jul 06 18:59:34.404406 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.404371 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-s259v\" (UID: \"2caf149a-9d45-4412-8c6f-006d28f81ef1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" Jul 06 18:59:34.404570 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.404439 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-rzpjn\" (UID: \"89edb5ad-301f-4971-a47a-82f8078b0c21\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn" Jul 06 18:59:34.404570 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.404499 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:34.404570 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.404545 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:34.404747 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:34.404570 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 18:59:34.404747 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.404587 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls\") pod \"dns-default-mkqz8\" (UID: \"41eea561-3913-46d4-beca-714c42bc07cd\") " pod="openshift-dns/dns-default-mkqz8" Jul 06 18:59:34.404747 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:34.404608 2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 18:59:34.404747 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:34.404662 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls podName:2caf149a-9d45-4412-8c6f-006d28f81ef1 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:35.40464034 +0000 UTC m=+36.673822685 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-s259v" (UID: "2caf149a-9d45-4412-8c6f-006d28f81ef1") : secret "cluster-monitoring-operator-tls" not found Jul 06 18:59:34.404747 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:34.404672 2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 18:59:34.404747 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:34.404693 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls podName:89edb5ad-301f-4971-a47a-82f8078b0c21 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:35.40468196 +0000 UTC m=+36.673864283 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-rzpjn" (UID: "89edb5ad-301f-4971-a47a-82f8078b0c21") : secret "samples-operator-tls" not found Jul 06 18:59:34.404747 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:34.404701 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 18:59:34.404747 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:34.404717 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle podName:b78c33c0-d4f4-4a97-8621-90904852db08 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:35.404701651 +0000 UTC m=+36.673883977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle") pod "router-default-7456bbb6dc-zfg4t" (UID: "b78c33c0-d4f4-4a97-8621-90904852db08") : configmap references non-existent config key: service-ca.crt Jul 06 18:59:34.404747 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:34.404733 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls podName:41eea561-3913-46d4-beca-714c42bc07cd nodeName:}" failed. No retries permitted until 2026-07-06 18:59:35.404723914 +0000 UTC m=+36.673906235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls") pod "dns-default-mkqz8" (UID: "41eea561-3913-46d4-beca-714c42bc07cd") : secret "dns-default-metrics-tls" not found Jul 06 18:59:34.404747 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:34.404747 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs podName:b78c33c0-d4f4-4a97-8621-90904852db08 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:35.404739511 +0000 UTC m=+36.673921835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs") pod "router-default-7456bbb6dc-zfg4t" (UID: "b78c33c0-d4f4-4a97-8621-90904852db08") : secret "router-metrics-certs-default" not found Jul 06 18:59:34.599834 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.599744 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sc9lq" event={"ID":"d11343a5-a7ee-47e9-8875-d6e7ae822930","Type":"ContainerStarted","Data":"7dd9be617ef6e3eb2d62af6916fdb16840dd544f4e17ff28d790f550ced29016"} Jul 06 18:59:34.601102 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.601069 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" event={"ID":"1923cd02-24e0-4753-be72-cb22fbd5c05f","Type":"ContainerStarted","Data":"021ec36f1a2b086df4f84c307ad347321326b8acf0353190ba0ba0bb0b64a490"} Jul 06 18:59:34.602360 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.602321 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-jjlwh" event={"ID":"0c10df95-ff62-4f94-a74d-91742dc4c92c","Type":"ContainerStarted","Data":"32e719b05a5dd190bf26b1099b0a7b849583690e93166d7c42f749a75c4a9822"} Jul 06 18:59:34.603654 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.603631 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" event={"ID":"724d10df-ec14-4dcd-8b9c-5265acf5ed80","Type":"ContainerStarted","Data":"d83541a364be4022d878ef2783bc708c92192a470663a974195f39020b301e18"} Jul 06 18:59:34.604776 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.604754 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-5t7tr" event={"ID":"1962c753-8b1d-4ac7-9e03-9756f80ef241","Type":"ContainerStarted","Data":"6be391f5faeedbd871595668f7230faadd86b1b9b0305dddd067ba082cd3e33d"} Jul 06 18:59:34.605992 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.605967 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s8v9j" event={"ID":"628f818d-2286-4af1-b946-5779244f0bff","Type":"ContainerStarted","Data":"0723952afab54bafed618c730ad62c5a929e0d421750fc69bdf4ddb46155ae65"} Jul 06 18:59:34.607245 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:34.607214 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b4bv7" event={"ID":"13153a33-8136-473a-b96d-7bd6b1e8c5e3","Type":"ContainerStarted","Data":"8c44eed79fbca3d1baefb5339d7a11d6e96f910f95c1a65387b55aa01147fe78"} Jul 06 18:59:35.279920 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:35.279892 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 18:59:35.284125 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:35.283925 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kw7z\"" Jul 06 18:59:35.284125 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:35.283995 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 18:59:35.311381 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:35.310699 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:35.311381 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:35.310948 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 18:59:35.311381 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:35.310965 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b6b7754b-gx4l2: secret "image-registry-tls" not found Jul 06 18:59:35.311381 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:35.311024 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls podName:5c9d5b1a-9817-4d71-a7b4-ff6ced34939c nodeName:}" failed. No retries permitted until 2026-07-06 18:59:37.311004303 +0000 UTC m=+38.580186628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls") pod "image-registry-57b6b7754b-gx4l2" (UID: "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c") : secret "image-registry-tls" not found Jul 06 18:59:35.413241 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:35.411576 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-s259v\" (UID: \"2caf149a-9d45-4412-8c6f-006d28f81ef1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" Jul 06 18:59:35.413241 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:35.411633 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-rzpjn\" (UID: \"89edb5ad-301f-4971-a47a-82f8078b0c21\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn" Jul 06 18:59:35.413241 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:35.411705 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:35.413241 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:35.411752 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:35.413241 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:35.411814 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls\") pod \"dns-default-mkqz8\" (UID: \"41eea561-3913-46d4-beca-714c42bc07cd\") " pod="openshift-dns/dns-default-mkqz8" Jul 06 18:59:35.413241 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:35.411950 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 18:59:35.413241 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:35.412013 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls podName:41eea561-3913-46d4-beca-714c42bc07cd nodeName:}" failed. No retries permitted until 2026-07-06 18:59:37.411994465 +0000 UTC m=+38.681176792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls") pod "dns-default-mkqz8" (UID: "41eea561-3913-46d4-beca-714c42bc07cd") : secret "dns-default-metrics-tls" not found Jul 06 18:59:35.413241 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:35.412403 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 18:59:35.413241 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:35.412478 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls podName:2caf149a-9d45-4412-8c6f-006d28f81ef1 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:37.412460603 +0000 UTC m=+38.681642932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-s259v" (UID: "2caf149a-9d45-4412-8c6f-006d28f81ef1") : secret "cluster-monitoring-operator-tls" not found Jul 06 18:59:35.413241 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:35.412540 2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 18:59:35.413241 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:35.412572 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls podName:89edb5ad-301f-4971-a47a-82f8078b0c21 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:37.412560906 +0000 UTC m=+38.681743228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-rzpjn" (UID: "89edb5ad-301f-4971-a47a-82f8078b0c21") : secret "samples-operator-tls" not found Jul 06 18:59:35.413241 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:35.412622 2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 18:59:35.413241 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:35.412657 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs podName:b78c33c0-d4f4-4a97-8621-90904852db08 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:37.412644181 +0000 UTC m=+38.681826504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs") pod "router-default-7456bbb6dc-zfg4t" (UID: "b78c33c0-d4f4-4a97-8621-90904852db08") : secret "router-metrics-certs-default" not found Jul 06 18:59:35.413241 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:35.412719 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle podName:b78c33c0-d4f4-4a97-8621-90904852db08 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:37.412709282 +0000 UTC m=+38.681891603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle") pod "router-default-7456bbb6dc-zfg4t" (UID: "b78c33c0-d4f4-4a97-8621-90904852db08") : configmap references non-existent config key: service-ca.crt Jul 06 18:59:36.626397 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:36.626358 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b4bv7" event={"ID":"13153a33-8136-473a-b96d-7bd6b1e8c5e3","Type":"ContainerStarted","Data":"5197a211379cfa8a74004a12bc82a2c977d58d9bf5d4e90bb398e4aa11544bf6"} Jul 06 18:59:36.958303 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:36.958262 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gm25_99098e94-f6ae-4918-b007-e85999dfe259/konnectivity-agent/0.log" limit=1024 Jul 06 18:59:37.057902 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:37.057865 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gm25_99098e94-f6ae-4918-b007-e85999dfe259/konnectivity-agent/0.log" limit=1024 Jul 06 18:59:37.332923 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:37.332879 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:37.333121 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:37.333109 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 18:59:37.333182 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:37.333126 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b6b7754b-gx4l2: secret "image-registry-tls" not found Jul 06 18:59:37.333218 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:37.333183 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls podName:5c9d5b1a-9817-4d71-a7b4-ff6ced34939c nodeName:}" failed. No retries permitted until 2026-07-06 18:59:41.333165826 +0000 UTC m=+42.602348150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls") pod "image-registry-57b6b7754b-gx4l2" (UID: "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c") : secret "image-registry-tls" not found Jul 06 18:59:37.435170 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:37.434732 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:37.435170 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:37.434799 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:37.435170 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:37.434851 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls\") pod \"dns-default-mkqz8\" (UID: \"41eea561-3913-46d4-beca-714c42bc07cd\") " pod="openshift-dns/dns-default-mkqz8" Jul 06 18:59:37.435170 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:37.434912 2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 18:59:37.435170 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:37.434939 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-s259v\" (UID: \"2caf149a-9d45-4412-8c6f-006d28f81ef1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" Jul 06 18:59:37.435170 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:37.434974 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs podName:b78c33c0-d4f4-4a97-8621-90904852db08 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:41.434954429 +0000 UTC m=+42.704136774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs") pod "router-default-7456bbb6dc-zfg4t" (UID: "b78c33c0-d4f4-4a97-8621-90904852db08") : secret "router-metrics-certs-default" not found Jul 06 18:59:37.435170 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:37.434995 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle podName:b78c33c0-d4f4-4a97-8621-90904852db08 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:41.434987425 +0000 UTC m=+42.704169745 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle") pod "router-default-7456bbb6dc-zfg4t" (UID: "b78c33c0-d4f4-4a97-8621-90904852db08") : configmap references non-existent config key: service-ca.crt Jul 06 18:59:37.435170 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:37.435029 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-rzpjn\" (UID: \"89edb5ad-301f-4971-a47a-82f8078b0c21\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn" Jul 06 18:59:37.435170 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:37.435031 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 18:59:37.435170 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:37.435084 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 18:59:37.435170 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:37.435105 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls podName:2caf149a-9d45-4412-8c6f-006d28f81ef1 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:41.435094346 +0000 UTC m=+42.704276691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-s259v" (UID: "2caf149a-9d45-4412-8c6f-006d28f81ef1") : secret "cluster-monitoring-operator-tls" not found Jul 06 18:59:37.435170 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:37.435135 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls podName:41eea561-3913-46d4-beca-714c42bc07cd nodeName:}" failed. No retries permitted until 2026-07-06 18:59:41.435119497 +0000 UTC m=+42.704301841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls") pod "dns-default-mkqz8" (UID: "41eea561-3913-46d4-beca-714c42bc07cd") : secret "dns-default-metrics-tls" not found Jul 06 18:59:37.435959 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:37.435702 2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 18:59:37.435959 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:37.435767 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls podName:89edb5ad-301f-4971-a47a-82f8078b0c21 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:41.435752012 +0000 UTC m=+42.704934335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-rzpjn" (UID: "89edb5ad-301f-4971-a47a-82f8078b0c21") : secret "samples-operator-tls" not found Jul 06 18:59:37.652666 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:37.652564 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-b4bv7" podStartSLOduration=16.472084435 podStartE2EDuration="18.652547624s" totalImagesPullingTime="2.180463189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:59:37.651892376 +0000 UTC m=+38.921074719" watchObservedRunningTime="2026-07-06 18:59:37.652547624 +0000 UTC m=+38.921729996" Jul 06 18:59:37.757456 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:37.757403 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gm25_99098e94-f6ae-4918-b007-e85999dfe259/konnectivity-agent/0.log" limit=1024 Jul 06 18:59:37.863978 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:37.863927 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gm25_99098e94-f6ae-4918-b007-e85999dfe259/konnectivity-agent/0.log" limit=1024 Jul 06 18:59:38.101584 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:38.101550 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gm25_99098e94-f6ae-4918-b007-e85999dfe259/konnectivity-agent/0.log" limit=1024 Jul 06 18:59:38.210454 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:38.210348 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gm25_99098e94-f6ae-4918-b007-e85999dfe259/konnectivity-agent/0.log" limit=1024 Jul 06 18:59:39.044492 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:39.044213 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gm25_99098e94-f6ae-4918-b007-e85999dfe259/konnectivity-agent/0.log" limit=1024 Jul 06 18:59:39.118371 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:39.118325 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gm25_99098e94-f6ae-4918-b007-e85999dfe259/konnectivity-agent/0.log" limit=1024 Jul 06 18:59:39.905714 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:39.905675 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gm25_99098e94-f6ae-4918-b007-e85999dfe259/konnectivity-agent/0.log" limit=1024 Jul 06 18:59:40.066877 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:40.066845 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gm25_99098e94-f6ae-4918-b007-e85999dfe259/konnectivity-agent/0.log" limit=1024 Jul 06 18:59:40.592855 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:40.592769 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gm25_99098e94-f6ae-4918-b007-e85999dfe259/konnectivity-agent/0.log" limit=1024 Jul 06 18:59:40.688598 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:40.688567 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gm25_99098e94-f6ae-4918-b007-e85999dfe259/konnectivity-agent/0.log" limit=1024 Jul 06 18:59:40.789961 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:40.789924 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gm25_99098e94-f6ae-4918-b007-e85999dfe259/konnectivity-agent/0.log" limit=1024 Jul 06 18:59:41.371276 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:41.371229 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:41.371738 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:41.371394 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 18:59:41.371738 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:41.371408 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b6b7754b-gx4l2: secret "image-registry-tls" not found Jul 06 18:59:41.371738 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:41.371502 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls podName:5c9d5b1a-9817-4d71-a7b4-ff6ced34939c nodeName:}" failed. No retries permitted until 2026-07-06 18:59:49.371484538 +0000 UTC m=+50.640666861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls") pod "image-registry-57b6b7754b-gx4l2" (UID: "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c") : secret "image-registry-tls" not found Jul 06 18:59:41.472014 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:41.471947 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls\") pod \"dns-default-mkqz8\" (UID: \"41eea561-3913-46d4-beca-714c42bc07cd\") " pod="openshift-dns/dns-default-mkqz8" Jul 06 18:59:41.472204 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:41.472076 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-s259v\" (UID: \"2caf149a-9d45-4412-8c6f-006d28f81ef1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" Jul 06 18:59:41.472204 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:41.472103 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-rzpjn\" (UID: \"89edb5ad-301f-4971-a47a-82f8078b0c21\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn" Jul 06 18:59:41.472204 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:41.472128 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 18:59:41.472204 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:41.472167 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:41.472204 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:41.472202 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls podName:41eea561-3913-46d4-beca-714c42bc07cd nodeName:}" failed. No retries permitted until 2026-07-06 18:59:49.47218241 +0000 UTC m=+50.741364759 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls") pod "dns-default-mkqz8" (UID: "41eea561-3913-46d4-beca-714c42bc07cd") : secret "dns-default-metrics-tls" not found Jul 06 18:59:41.472510 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:41.472244 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:41.472510 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:41.472250 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 18:59:41.472510 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:41.472344 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls podName:2caf149a-9d45-4412-8c6f-006d28f81ef1 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:49.472331252 +0000 UTC m=+50.741513589 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-s259v" (UID: "2caf149a-9d45-4412-8c6f-006d28f81ef1") : secret "cluster-monitoring-operator-tls" not found Jul 06 18:59:41.472510 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:41.472305 2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 18:59:41.472510 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:41.472358 2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 18:59:41.472510 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:41.472378 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs podName:b78c33c0-d4f4-4a97-8621-90904852db08 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:49.472368262 +0000 UTC m=+50.741550614 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs") pod "router-default-7456bbb6dc-zfg4t" (UID: "b78c33c0-d4f4-4a97-8621-90904852db08") : secret "router-metrics-certs-default" not found Jul 06 18:59:41.472510 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:41.472407 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle podName:b78c33c0-d4f4-4a97-8621-90904852db08 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:49.472390516 +0000 UTC m=+50.741572861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle") pod "router-default-7456bbb6dc-zfg4t" (UID: "b78c33c0-d4f4-4a97-8621-90904852db08") : configmap references non-existent config key: service-ca.crt Jul 06 18:59:41.472510 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:41.472451 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls podName:89edb5ad-301f-4971-a47a-82f8078b0c21 nodeName:}" failed. No retries permitted until 2026-07-06 18:59:49.472438972 +0000 UTC m=+50.741621294 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-rzpjn" (UID: "89edb5ad-301f-4971-a47a-82f8078b0c21") : secret "samples-operator-tls" not found Jul 06 18:59:44.651576 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:44.651225 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-5t7tr" event={"ID":"1962c753-8b1d-4ac7-9e03-9756f80ef241","Type":"ContainerStarted","Data":"03bd26f1b0ab14778593330b14d27ec0d982f23464bb876d70c40dfe36358e9a"} Jul 06 18:59:44.652860 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:44.652808 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s8v9j" event={"ID":"628f818d-2286-4af1-b946-5779244f0bff","Type":"ContainerStarted","Data":"4c2f856d91adeab397e21a63ba478e97c67a2469597f3a095362fd3e36c0ca74"} Jul 06 18:59:44.655789 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:44.655766 2553 generic.go:356] "Generic (PLEG): container finished" podID="27f875a5-8c79-42e8-b05f-819fe5dcc452" containerID="34d8b1e3dcd519cbfeb99d1f770366a0ef58a314250d8a0bc142914b7cc241d5" exitCode=0 Jul 06 18:59:44.655911 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:44.655861 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlq9c" event={"ID":"27f875a5-8c79-42e8-b05f-819fe5dcc452","Type":"ContainerDied","Data":"34d8b1e3dcd519cbfeb99d1f770366a0ef58a314250d8a0bc142914b7cc241d5"} Jul 06 18:59:44.658921 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:44.658895 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sc9lq" event={"ID":"d11343a5-a7ee-47e9-8875-d6e7ae822930","Type":"ContainerStarted","Data":"8ea604c48581a890f2f4d2aefa46e5a3df49fdbb9617d87307586c9d90fce507"} Jul 06 18:59:44.660867 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:44.660850 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/0.log" Jul 06 18:59:44.660939 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:44.660879 2553 generic.go:356] "Generic (PLEG): container finished" podID="1923cd02-24e0-4753-be72-cb22fbd5c05f" containerID="5f3d95faae7d75cafda20575a5de8110d00609666b58add8a6e263748dbdeffd" exitCode=255 Jul 06 18:59:44.660939 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:44.660930 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" event={"ID":"1923cd02-24e0-4753-be72-cb22fbd5c05f","Type":"ContainerDied","Data":"5f3d95faae7d75cafda20575a5de8110d00609666b58add8a6e263748dbdeffd"} Jul 06 18:59:44.661159 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:44.661143 2553 scope.go:117] "RemoveContainer" containerID="5f3d95faae7d75cafda20575a5de8110d00609666b58add8a6e263748dbdeffd" Jul 06 18:59:44.664106 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:44.664059 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-jjlwh" event={"ID":"0c10df95-ff62-4f94-a74d-91742dc4c92c","Type":"ContainerStarted","Data":"39ad1bee409b5bc80a166d81241d5854a2ce417976d35c7a18ab5d070d113e67"} Jul 06 18:59:44.665470 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:44.665450 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" event={"ID":"724d10df-ec14-4dcd-8b9c-5265acf5ed80","Type":"ContainerStarted","Data":"37185797dcb4b94dc3017011418a3e2fb4a9e37ff196087852b1e2f17af6fd8a"} Jul 06 18:59:44.742711 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:44.742658 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-5t7tr" podStartSLOduration=15.930013225 podStartE2EDuration="25.742638537s" totalImagesPullingTime="9.812625312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:59:44.681958101 +0000 UTC m=+45.951140442" watchObservedRunningTime="2026-07-06 18:59:44.742638537 +0000 UTC m=+46.011820880" Jul 06 18:59:44.769634 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:44.769582 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s8v9j" podStartSLOduration=15.975360808 podStartE2EDuration="25.769561428s" totalImagesPullingTime="9.79420062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:59:44.769448459 +0000 UTC m=+46.038630803" watchObservedRunningTime="2026-07-06 18:59:44.769561428 +0000 UTC m=+46.038743751" Jul 06 18:59:44.840822 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:44.837403 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sc9lq" podStartSLOduration=2.053846396 podStartE2EDuration="11.837379989s" totalImagesPullingTime="9.783533593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:59:44.809773341 +0000 UTC m=+46.078955685" watchObservedRunningTime="2026-07-06 18:59:44.837379989 +0000 UTC m=+46.106562331" Jul 06 18:59:44.867000 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:44.866948 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" podStartSLOduration=15.94215249 podStartE2EDuration="25.866927941s" totalImagesPullingTime="9.924775451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:59:44.838137939 +0000 UTC m=+46.107320282" watchObservedRunningTime="2026-07-06 18:59:44.866927941 +0000 UTC m=+46.136110285" Jul 06 18:59:44.884165 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:44.884113 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-jjlwh" podStartSLOduration=16.000983612 podStartE2EDuration="25.884094035s" totalImagesPullingTime="9.883110423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:59:44.883017753 +0000 UTC m=+46.152200095" watchObservedRunningTime="2026-07-06 18:59:44.884094035 +0000 UTC m=+46.153276378" Jul 06 18:59:45.455175 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.455096 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s9tch"] Jul 06 18:59:45.472299 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.472274 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:45.475831 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.475804 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 18:59:45.475963 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.475805 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g47v9\"" Jul 06 18:59:45.475963 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.475856 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 18:59:45.484460 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.484437 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s9tch"] Jul 06 18:59:45.518064 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.518034 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b345cada-d7a4-4b8f-af05-29d6b7f91edc-data-volume\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:45.518233 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.518129 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b345cada-d7a4-4b8f-af05-29d6b7f91edc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:45.518233 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.518159 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b345cada-d7a4-4b8f-af05-29d6b7f91edc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:45.518233 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.518183 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4z4c\" (UniqueName: \"kubernetes.io/projected/b345cada-d7a4-4b8f-af05-29d6b7f91edc-kube-api-access-g4z4c\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:45.518233 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.518205 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b345cada-d7a4-4b8f-af05-29d6b7f91edc-crio-socket\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:45.618938 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.618892 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b345cada-d7a4-4b8f-af05-29d6b7f91edc-data-volume\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:45.619098 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.619041 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b345cada-d7a4-4b8f-af05-29d6b7f91edc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:45.619098 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.619085 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b345cada-d7a4-4b8f-af05-29d6b7f91edc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:45.619227 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.619117 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4z4c\" (UniqueName: \"kubernetes.io/projected/b345cada-d7a4-4b8f-af05-29d6b7f91edc-kube-api-access-g4z4c\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:45.619227 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.619145 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b345cada-d7a4-4b8f-af05-29d6b7f91edc-crio-socket\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:45.619227 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:45.619209 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 18:59:45.619369 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:45.619282 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b345cada-d7a4-4b8f-af05-29d6b7f91edc-insights-runtime-extractor-tls podName:b345cada-d7a4-4b8f-af05-29d6b7f91edc nodeName:}" failed. No retries permitted until 2026-07-06 18:59:46.119261001 +0000 UTC m=+47.388443344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b345cada-d7a4-4b8f-af05-29d6b7f91edc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s9tch" (UID: "b345cada-d7a4-4b8f-af05-29d6b7f91edc") : secret "insights-runtime-extractor-tls" not found Jul 06 18:59:45.619369 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.619279 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b345cada-d7a4-4b8f-af05-29d6b7f91edc-data-volume\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:45.619369 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.619288 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b345cada-d7a4-4b8f-af05-29d6b7f91edc-crio-socket\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:45.619369 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:45.619338 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 18:59:45.619600 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:45.619396 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b345cada-d7a4-4b8f-af05-29d6b7f91edc-kube-rbac-proxy-cm podName:b345cada-d7a4-4b8f-af05-29d6b7f91edc nodeName:}" failed. No retries permitted until 2026-07-06 18:59:46.119378781 +0000 UTC m=+47.388561105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b345cada-d7a4-4b8f-af05-29d6b7f91edc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s9tch" (UID: "b345cada-d7a4-4b8f-af05-29d6b7f91edc") : configmap "kube-rbac-proxy" not found Jul 06 18:59:45.630240 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.630208 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4z4c\" (UniqueName: \"kubernetes.io/projected/b345cada-d7a4-4b8f-af05-29d6b7f91edc-kube-api-access-g4z4c\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:45.670603 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.670572 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 18:59:45.671004 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.670989 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/0.log" Jul 06 18:59:45.671055 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.671028 2553 generic.go:356] "Generic (PLEG): container finished" podID="1923cd02-24e0-4753-be72-cb22fbd5c05f" containerID="5d4e7d3dd010f87fcf8f9b065746ec7f71359065f25848d42e6b04558b3b9b58" exitCode=255 Jul 06 18:59:45.671143 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.671121 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" event={"ID":"1923cd02-24e0-4753-be72-cb22fbd5c05f","Type":"ContainerDied","Data":"5d4e7d3dd010f87fcf8f9b065746ec7f71359065f25848d42e6b04558b3b9b58"} Jul 06 18:59:45.671218 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.671204 2553 scope.go:117] "RemoveContainer" containerID="5f3d95faae7d75cafda20575a5de8110d00609666b58add8a6e263748dbdeffd" Jul 06 18:59:45.671439 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.671374 2553 scope.go:117] "RemoveContainer" containerID="5d4e7d3dd010f87fcf8f9b065746ec7f71359065f25848d42e6b04558b3b9b58" Jul 06 18:59:45.671637 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:45.671619 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-8m9wb_openshift-console-operator(1923cd02-24e0-4753-be72-cb22fbd5c05f)\"" pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" podUID="1923cd02-24e0-4753-be72-cb22fbd5c05f" Jul 06 18:59:45.674327 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.674305 2553 generic.go:356] "Generic (PLEG): container finished" podID="27f875a5-8c79-42e8-b05f-819fe5dcc452" containerID="e7662b3826a2f50679c4d34e30225cd214c7a22487034b0eb6d14596fbce74bf" exitCode=0 Jul 06 18:59:45.674454 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:45.674408 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlq9c" event={"ID":"27f875a5-8c79-42e8-b05f-819fe5dcc452","Type":"ContainerDied","Data":"e7662b3826a2f50679c4d34e30225cd214c7a22487034b0eb6d14596fbce74bf"} Jul 06 18:59:46.125798 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:46.125759 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b345cada-d7a4-4b8f-af05-29d6b7f91edc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:46.125975 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:46.125818 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b345cada-d7a4-4b8f-af05-29d6b7f91edc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:46.125975 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:46.125931 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 18:59:46.125975 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:46.125957 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 18:59:46.126130 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:46.126024 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b345cada-d7a4-4b8f-af05-29d6b7f91edc-insights-runtime-extractor-tls podName:b345cada-d7a4-4b8f-af05-29d6b7f91edc nodeName:}" failed. No retries permitted until 2026-07-06 18:59:47.12599546 +0000 UTC m=+48.395177792 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b345cada-d7a4-4b8f-af05-29d6b7f91edc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s9tch" (UID: "b345cada-d7a4-4b8f-af05-29d6b7f91edc") : secret "insights-runtime-extractor-tls" not found Jul 06 18:59:46.126130 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:46.126044 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b345cada-d7a4-4b8f-af05-29d6b7f91edc-kube-rbac-proxy-cm podName:b345cada-d7a4-4b8f-af05-29d6b7f91edc nodeName:}" failed. No retries permitted until 2026-07-06 18:59:47.126034326 +0000 UTC m=+48.395216650 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b345cada-d7a4-4b8f-af05-29d6b7f91edc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s9tch" (UID: "b345cada-d7a4-4b8f-af05-29d6b7f91edc") : configmap "kube-rbac-proxy" not found Jul 06 18:59:46.681675 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:46.681643 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vlq9c" event={"ID":"27f875a5-8c79-42e8-b05f-819fe5dcc452","Type":"ContainerStarted","Data":"297673fab2c2a3ad43e994bf9d29513b56abe36ac3748d2cd5a1488b3661771d"} Jul 06 18:59:46.683089 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:46.683065 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 18:59:46.683340 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:46.683326 2553 scope.go:117] "RemoveContainer" containerID="5d4e7d3dd010f87fcf8f9b065746ec7f71359065f25848d42e6b04558b3b9b58" Jul 06 18:59:46.683498 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:46.683480 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-8m9wb_openshift-console-operator(1923cd02-24e0-4753-be72-cb22fbd5c05f)\"" pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" podUID="1923cd02-24e0-4753-be72-cb22fbd5c05f" Jul 06 18:59:46.706127 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:46.706082 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vlq9c" podStartSLOduration=6.898570412 podStartE2EDuration="47.70606764s" totalImagesPullingTime="40.807497228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:58:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 18:59:46.704715605 +0000 UTC m=+47.973897947" watchObservedRunningTime="2026-07-06 18:59:46.70606764 +0000 UTC m=+47.975249998" Jul 06 18:59:47.025071 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:47.025042 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nb9d8_2f70581d-708c-43ae-871a-a126644cefff/dns-node-resolver/0.log" Jul 06 18:59:47.135106 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:47.135069 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b345cada-d7a4-4b8f-af05-29d6b7f91edc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:47.135293 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:47.135110 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b345cada-d7a4-4b8f-af05-29d6b7f91edc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:47.135293 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:47.135187 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 18:59:47.135293 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:47.135241 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b345cada-d7a4-4b8f-af05-29d6b7f91edc-kube-rbac-proxy-cm podName:b345cada-d7a4-4b8f-af05-29d6b7f91edc nodeName:}" failed. No retries permitted until 2026-07-06 18:59:49.135227447 +0000 UTC m=+50.404409767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b345cada-d7a4-4b8f-af05-29d6b7f91edc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s9tch" (UID: "b345cada-d7a4-4b8f-af05-29d6b7f91edc") : configmap "kube-rbac-proxy" not found Jul 06 18:59:47.135293 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:47.135238 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 18:59:47.135486 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:47.135302 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b345cada-d7a4-4b8f-af05-29d6b7f91edc-insights-runtime-extractor-tls podName:b345cada-d7a4-4b8f-af05-29d6b7f91edc nodeName:}" failed. No retries permitted until 2026-07-06 18:59:49.135284796 +0000 UTC m=+50.404467137 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b345cada-d7a4-4b8f-af05-29d6b7f91edc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s9tch" (UID: "b345cada-d7a4-4b8f-af05-29d6b7f91edc") : secret "insights-runtime-extractor-tls" not found Jul 06 18:59:48.027461 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:48.027428 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v2xjc_c996c2ca-cb6f-420e-b4d0-5b9720137b7d/node-ca/0.log" Jul 06 18:59:48.832841 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:48.832804 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sc9lq_d11343a5-a7ee-47e9-8875-d6e7ae822930/serve-healthcheck-canary/0.log" Jul 06 18:59:49.151996 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:49.151904 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b345cada-d7a4-4b8f-af05-29d6b7f91edc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:49.151996 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:49.151949 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b345cada-d7a4-4b8f-af05-29d6b7f91edc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:49.152386 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:49.152031 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 18:59:49.152386 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:49.152065 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 18:59:49.152386 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:49.152088 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b345cada-d7a4-4b8f-af05-29d6b7f91edc-kube-rbac-proxy-cm podName:b345cada-d7a4-4b8f-af05-29d6b7f91edc nodeName:}" failed. No retries permitted until 2026-07-06 18:59:53.152074204 +0000 UTC m=+54.421256523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b345cada-d7a4-4b8f-af05-29d6b7f91edc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s9tch" (UID: "b345cada-d7a4-4b8f-af05-29d6b7f91edc") : configmap "kube-rbac-proxy" not found Jul 06 18:59:49.152386 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:49.152117 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b345cada-d7a4-4b8f-af05-29d6b7f91edc-insights-runtime-extractor-tls podName:b345cada-d7a4-4b8f-af05-29d6b7f91edc nodeName:}" failed. No retries permitted until 2026-07-06 18:59:53.15210206 +0000 UTC m=+54.421284396 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b345cada-d7a4-4b8f-af05-29d6b7f91edc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s9tch" (UID: "b345cada-d7a4-4b8f-af05-29d6b7f91edc") : secret "insights-runtime-extractor-tls" not found Jul 06 18:59:49.454595 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:49.454506 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 18:59:49.454728 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:49.454655 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 18:59:49.454728 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:49.454674 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57b6b7754b-gx4l2: secret "image-registry-tls" not found Jul 06 18:59:49.454810 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:49.454731 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls podName:5c9d5b1a-9817-4d71-a7b4-ff6ced34939c nodeName:}" failed. No retries permitted until 2026-07-06 19:00:05.45471545 +0000 UTC m=+66.723897817 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls") pod "image-registry-57b6b7754b-gx4l2" (UID: "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c") : secret "image-registry-tls" not found Jul 06 18:59:49.555527 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:49.555493 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:49.555715 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:49.555540 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 18:59:49.555715 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:49.555569 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls\") pod \"dns-default-mkqz8\" (UID: \"41eea561-3913-46d4-beca-714c42bc07cd\") " pod="openshift-dns/dns-default-mkqz8" Jul 06 18:59:49.555715 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:49.555673 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 18:59:49.555715 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:49.555682 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle podName:b78c33c0-d4f4-4a97-8621-90904852db08 nodeName:}" failed. No retries permitted until 2026-07-06 19:00:05.555665434 +0000 UTC m=+66.824847774 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle") pod "router-default-7456bbb6dc-zfg4t" (UID: "b78c33c0-d4f4-4a97-8621-90904852db08") : configmap references non-existent config key: service-ca.crt Jul 06 18:59:49.555715 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:49.555673 2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 18:59:49.555973 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:49.555708 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls podName:41eea561-3913-46d4-beca-714c42bc07cd nodeName:}" failed. No retries permitted until 2026-07-06 19:00:05.555698998 +0000 UTC m=+66.824881323 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls") pod "dns-default-mkqz8" (UID: "41eea561-3913-46d4-beca-714c42bc07cd") : secret "dns-default-metrics-tls" not found Jul 06 18:59:49.555973 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:49.555778 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs podName:b78c33c0-d4f4-4a97-8621-90904852db08 nodeName:}" failed. No retries permitted until 2026-07-06 19:00:05.555760005 +0000 UTC m=+66.824942330 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs") pod "router-default-7456bbb6dc-zfg4t" (UID: "b78c33c0-d4f4-4a97-8621-90904852db08") : secret "router-metrics-certs-default" not found Jul 06 18:59:49.555973 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:49.555862 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-s259v\" (UID: \"2caf149a-9d45-4412-8c6f-006d28f81ef1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" Jul 06 18:59:49.555973 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:49.555899 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-rzpjn\" (UID: \"89edb5ad-301f-4971-a47a-82f8078b0c21\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn" Jul 06 18:59:49.556150 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:49.555997 2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 18:59:49.556150 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:49.556036 2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 18:59:49.556150 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:49.556043 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls podName:2caf149a-9d45-4412-8c6f-006d28f81ef1 nodeName:}" failed. No retries permitted until 2026-07-06 19:00:05.556031451 +0000 UTC m=+66.825213772 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-s259v" (UID: "2caf149a-9d45-4412-8c6f-006d28f81ef1") : secret "cluster-monitoring-operator-tls" not found Jul 06 18:59:49.556150 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:49.556074 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls podName:89edb5ad-301f-4971-a47a-82f8078b0c21 nodeName:}" failed. No retries permitted until 2026-07-06 19:00:05.556062375 +0000 UTC m=+66.825244698 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-rzpjn" (UID: "89edb5ad-301f-4971-a47a-82f8078b0c21") : secret "samples-operator-tls" not found Jul 06 18:59:53.186096 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:53.186053 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b345cada-d7a4-4b8f-af05-29d6b7f91edc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:53.186096 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:53.186099 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b345cada-d7a4-4b8f-af05-29d6b7f91edc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:53.186864 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:53.186246 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 18:59:53.186864 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:53.186373 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b345cada-d7a4-4b8f-af05-29d6b7f91edc-insights-runtime-extractor-tls podName:b345cada-d7a4-4b8f-af05-29d6b7f91edc nodeName:}" failed. No retries permitted until 2026-07-06 19:00:01.186348873 +0000 UTC m=+62.455531210 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b345cada-d7a4-4b8f-af05-29d6b7f91edc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s9tch" (UID: "b345cada-d7a4-4b8f-af05-29d6b7f91edc") : secret "insights-runtime-extractor-tls" not found Jul 06 18:59:53.186864 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:53.186603 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b345cada-d7a4-4b8f-af05-29d6b7f91edc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 18:59:53.990371 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:53.990333 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" Jul 06 18:59:53.990570 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:53.990444 2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" Jul 06 18:59:53.990741 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:53.990729 2553 scope.go:117] "RemoveContainer" containerID="5d4e7d3dd010f87fcf8f9b065746ec7f71359065f25848d42e6b04558b3b9b58" Jul 06 18:59:53.990896 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:53.990878 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-8m9wb_openshift-console-operator(1923cd02-24e0-4753-be72-cb22fbd5c05f)\"" pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" podUID="1923cd02-24e0-4753-be72-cb22fbd5c05f" Jul 06 18:59:54.708732 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:54.708707 2553 scope.go:117] "RemoveContainer" containerID="5d4e7d3dd010f87fcf8f9b065746ec7f71359065f25848d42e6b04558b3b9b58" Jul 06 18:59:54.709107 ip-10-0-141-237 kubenswrapper[2553]: E0706 18:59:54.708861 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-8m9wb_openshift-console-operator(1923cd02-24e0-4753-be72-cb22fbd5c05f)\"" pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" podUID="1923cd02-24e0-4753-be72-cb22fbd5c05f" Jul 06 18:59:59.599900 ip-10-0-141-237 kubenswrapper[2553]: I0706 18:59:59.599871 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x5g5z" Jul 06 19:00:01.259405 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:01.259365 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b345cada-d7a4-4b8f-af05-29d6b7f91edc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 19:00:01.261887 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:01.261861 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b345cada-d7a4-4b8f-af05-29d6b7f91edc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s9tch\" (UID: \"b345cada-d7a4-4b8f-af05-29d6b7f91edc\") " pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 19:00:01.384224 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:01.384192 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g47v9\"" Jul 06 19:00:01.393013 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:01.392989 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s9tch" Jul 06 19:00:01.523136 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:01.523056 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s9tch"] Jul 06 19:00:01.527531 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:00:01.527507 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb345cada_d7a4_4b8f_af05_29d6b7f91edc.slice/crio-7bdc9bed867fd26e8afa05f3c62c20efa837a0d9fa41afe629b94617d90af71e WatchSource:0}: Error finding container 7bdc9bed867fd26e8afa05f3c62c20efa837a0d9fa41afe629b94617d90af71e: Status 404 returned error can't find the container with id 7bdc9bed867fd26e8afa05f3c62c20efa837a0d9fa41afe629b94617d90af71e Jul 06 19:00:01.729127 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:01.729085 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s9tch" event={"ID":"b345cada-d7a4-4b8f-af05-29d6b7f91edc","Type":"ContainerStarted","Data":"7bdc9bed867fd26e8afa05f3c62c20efa837a0d9fa41afe629b94617d90af71e"} Jul 06 19:00:03.883942 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:03.883909 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs\") pod \"network-metrics-daemon-jmvjq\" (UID: \"e7b6d1cb-2542-4861-9475-25365c8a5d15\") " pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 19:00:03.884459 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:03.883995 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret\") pod \"global-pull-secret-syncer-v5xs7\" (UID: \"f7d2aeda-aa6f-4865-ade0-f9aa24ea785a\") " pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 19:00:03.886473 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:03.886450 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:00:03.886612 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:03.886450 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:00:03.897293 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:03.897267 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f7d2aeda-aa6f-4865-ade0-f9aa24ea785a-original-pull-secret\") pod \"global-pull-secret-syncer-v5xs7\" (UID: \"f7d2aeda-aa6f-4865-ade0-f9aa24ea785a\") " pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 19:00:03.897293 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:03.897286 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b6d1cb-2542-4861-9475-25365c8a5d15-metrics-certs\") pod \"network-metrics-daemon-jmvjq\" (UID: \"e7b6d1cb-2542-4861-9475-25365c8a5d15\") " pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 19:00:03.985278 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:03.985241 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmgrx\" (UniqueName: \"kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx\") pod \"network-check-target-hklfk\" (UID: \"aab0c2d2-909e-466d-bd99-c4f4b3830060\") " pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 19:00:03.988559 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:03.988530 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmgrx\" (UniqueName: \"kubernetes.io/projected/aab0c2d2-909e-466d-bd99-c4f4b3830060-kube-api-access-cmgrx\") pod \"network-check-target-hklfk\" (UID: \"aab0c2d2-909e-466d-bd99-c4f4b3830060\") " pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 19:00:03.990133 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:03.990108 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v5xs7" Jul 06 19:00:03.998478 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:03.998458 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zn9xt\"" Jul 06 19:00:04.007295 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:04.007270 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 19:00:04.095691 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:04.095653 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6kw7z\"" Jul 06 19:00:04.104277 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:04.104252 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmvjq" Jul 06 19:00:05.498185 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.498148 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 19:00:05.501014 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.500988 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls\") pod \"image-registry-57b6b7754b-gx4l2\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 19:00:05.599528 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.598845 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 19:00:05.599528 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.598904 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 19:00:05.599528 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.598947 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls\") pod \"dns-default-mkqz8\" (UID: \"41eea561-3913-46d4-beca-714c42bc07cd\") " pod="openshift-dns/dns-default-mkqz8" Jul 06 19:00:05.599528 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.599016 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-s259v\" (UID: \"2caf149a-9d45-4412-8c6f-006d28f81ef1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" Jul 06 19:00:05.599528 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.599047 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-rzpjn\" (UID: \"89edb5ad-301f-4971-a47a-82f8078b0c21\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn" Jul 06 19:00:05.602121 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.601382 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b78c33c0-d4f4-4a97-8621-90904852db08-service-ca-bundle\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 19:00:05.606286 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.604096 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/89edb5ad-301f-4971-a47a-82f8078b0c21-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-rzpjn\" (UID: \"89edb5ad-301f-4971-a47a-82f8078b0c21\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn" Jul 06 19:00:05.606286 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.605765 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41eea561-3913-46d4-beca-714c42bc07cd-metrics-tls\") pod \"dns-default-mkqz8\" (UID: \"41eea561-3913-46d4-beca-714c42bc07cd\") " pod="openshift-dns/dns-default-mkqz8" Jul 06 19:00:05.606286 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.605893 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2caf149a-9d45-4412-8c6f-006d28f81ef1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-s259v\" (UID: \"2caf149a-9d45-4412-8c6f-006d28f81ef1\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" Jul 06 19:00:05.607688 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.607432 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78c33c0-d4f4-4a97-8621-90904852db08-metrics-certs\") pod \"router-default-7456bbb6dc-zfg4t\" (UID: \"b78c33c0-d4f4-4a97-8621-90904852db08\") " pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 19:00:05.693102 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.693069 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jmvjq"] Jul 06 19:00:05.696780 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:00:05.696745 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b6d1cb_2542_4861_9475_25365c8a5d15.slice/crio-37e0dac83fbca7ba406b7dd3152e788473136a3cf8e3be296e9df50ff919a256 WatchSource:0}: Error finding container 37e0dac83fbca7ba406b7dd3152e788473136a3cf8e3be296e9df50ff919a256: Status 404 returned error can't find the container with id 37e0dac83fbca7ba406b7dd3152e788473136a3cf8e3be296e9df50ff919a256 Jul 06 19:00:05.708961 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.708931 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v5xs7"] Jul 06 19:00:05.711063 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:00:05.711034 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d2aeda_aa6f_4865_ade0_f9aa24ea785a.slice/crio-f4aaeb3263599c71459d9a1fac37df5cac18a554639541b121839ef7ad1987f1 WatchSource:0}: Error finding container f4aaeb3263599c71459d9a1fac37df5cac18a554639541b121839ef7ad1987f1: Status 404 returned error can't find the container with id f4aaeb3263599c71459d9a1fac37df5cac18a554639541b121839ef7ad1987f1 Jul 06 19:00:05.723846 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.723826 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ch228\"" Jul 06 19:00:05.729280 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.729265 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 19:00:05.742875 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.742846 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v5xs7" event={"ID":"f7d2aeda-aa6f-4865-ade0-f9aa24ea785a","Type":"ContainerStarted","Data":"f4aaeb3263599c71459d9a1fac37df5cac18a554639541b121839ef7ad1987f1"} Jul 06 19:00:05.743988 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.743964 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jmvjq" event={"ID":"e7b6d1cb-2542-4861-9475-25365c8a5d15","Type":"ContainerStarted","Data":"37e0dac83fbca7ba406b7dd3152e788473136a3cf8e3be296e9df50ff919a256"} Jul 06 19:00:05.745333 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.745312 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s9tch" event={"ID":"b345cada-d7a4-4b8f-af05-29d6b7f91edc","Type":"ContainerStarted","Data":"4c19265d7700a9b07cd9f66da89406f441d17475eb752e72c0c9f99fe71aa8bf"} Jul 06 19:00:05.757006 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.756991 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-z9hg5\"" Jul 06 19:00:05.763039 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.763016 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8l6kc\"" Jul 06 19:00:05.764774 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.764756 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" Jul 06 19:00:05.772022 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.772001 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 19:00:05.786625 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.786604 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hfrrk\"" Jul 06 19:00:05.795640 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.795616 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn" Jul 06 19:00:05.820007 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.819944 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2f7gj\"" Jul 06 19:00:05.831722 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.830145 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mkqz8" Jul 06 19:00:05.912004 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.911941 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hklfk"] Jul 06 19:00:05.914168 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.914100 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57b6b7754b-gx4l2"] Jul 06 19:00:05.958042 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.958020 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v"] Jul 06 19:00:05.964117 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:00:05.964080 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2caf149a_9d45_4412_8c6f_006d28f81ef1.slice/crio-8190a1565cd70181d28927583ae1e9c722124b62ad08a73f8edc3fd2fe47a0e3 WatchSource:0}: Error finding container 8190a1565cd70181d28927583ae1e9c722124b62ad08a73f8edc3fd2fe47a0e3: Status 404 returned error can't find the container with id 8190a1565cd70181d28927583ae1e9c722124b62ad08a73f8edc3fd2fe47a0e3 Jul 06 19:00:05.978843 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.978817 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7456bbb6dc-zfg4t"] Jul 06 19:00:05.985625 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:00:05.985031 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78c33c0_d4f4_4a97_8621_90904852db08.slice/crio-8d04400693c0782cd6bc84f2f8763b546ef8bdbd580d0ddf64dcc70af9cb771f WatchSource:0}: Error finding container 8d04400693c0782cd6bc84f2f8763b546ef8bdbd580d0ddf64dcc70af9cb771f: Status 404 returned error can't find the container with id 8d04400693c0782cd6bc84f2f8763b546ef8bdbd580d0ddf64dcc70af9cb771f Jul 06 19:00:05.997941 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:05.997913 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn"] Jul 06 19:00:06.037259 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:06.037233 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mkqz8"] Jul 06 19:00:06.042075 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:00:06.042048 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41eea561_3913_46d4_beca_714c42bc07cd.slice/crio-5011cb8164950b6334c43df82bbd2c868195da591c68662d4c370e54ff4b5b82 WatchSource:0}: Error finding container 5011cb8164950b6334c43df82bbd2c868195da591c68662d4c370e54ff4b5b82: Status 404 returned error can't find the container with id 5011cb8164950b6334c43df82bbd2c868195da591c68662d4c370e54ff4b5b82 Jul 06 19:00:06.750262 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:06.750226 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn" event={"ID":"89edb5ad-301f-4971-a47a-82f8078b0c21","Type":"ContainerStarted","Data":"1fb63d2881b6cf394af9087253c99a7a13a88de93568516a96e4c52838eef92f"} Jul 06 19:00:06.751623 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:06.751586 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" event={"ID":"2caf149a-9d45-4412-8c6f-006d28f81ef1","Type":"ContainerStarted","Data":"8190a1565cd70181d28927583ae1e9c722124b62ad08a73f8edc3fd2fe47a0e3"} Jul 06 19:00:06.753691 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:06.753628 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s9tch" event={"ID":"b345cada-d7a4-4b8f-af05-29d6b7f91edc","Type":"ContainerStarted","Data":"a31ce510accc8c16d204ddbd9517a58fc6e52d607933f0c196c6a48d0b82b2fb"} Jul 06 19:00:06.754925 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:06.754893 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mkqz8" event={"ID":"41eea561-3913-46d4-beca-714c42bc07cd","Type":"ContainerStarted","Data":"5011cb8164950b6334c43df82bbd2c868195da591c68662d4c370e54ff4b5b82"} Jul 06 19:00:06.756358 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:06.756335 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" event={"ID":"b78c33c0-d4f4-4a97-8621-90904852db08","Type":"ContainerStarted","Data":"97955b700a53e21696647a488b5830f8b3381bce9bb5272bc0320606a5851611"} Jul 06 19:00:06.756483 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:06.756376 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" event={"ID":"b78c33c0-d4f4-4a97-8621-90904852db08","Type":"ContainerStarted","Data":"8d04400693c0782cd6bc84f2f8763b546ef8bdbd580d0ddf64dcc70af9cb771f"} Jul 06 19:00:06.758020 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:06.757990 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" event={"ID":"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c","Type":"ContainerStarted","Data":"520d6e83f8246754eb9a4aaf6343acda1175b8a5a2d1f0a30668107869bd5684"} Jul 06 19:00:06.758127 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:06.758032 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" event={"ID":"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c","Type":"ContainerStarted","Data":"c8a25ac40a87f10a1eef4cebf1240c4ab64ae30df76a5678147a258f8dc6fda0"} Jul 06 19:00:06.758199 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:06.758129 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 19:00:06.759858 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:06.759838 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hklfk" event={"ID":"aab0c2d2-909e-466d-bd99-c4f4b3830060","Type":"ContainerStarted","Data":"74c9e2c4b2e479616833428adb33099ed304c970945ec884bb89ac93be50efd8"} Jul 06 19:00:06.759966 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:06.759869 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hklfk" event={"ID":"aab0c2d2-909e-466d-bd99-c4f4b3830060","Type":"ContainerStarted","Data":"969eb1dc9a5662e014245e87603a758b5c91c6dc763ab17b59bf3fe883ba12ae"} Jul 06 19:00:06.760033 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:06.759975 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 19:00:06.773016 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:06.772995 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 19:00:06.775582 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:06.775563 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 19:00:06.784817 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:06.784771 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" podStartSLOduration=47.784755612 podStartE2EDuration="47.784755612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:00:06.783165195 +0000 UTC m=+68.052347538" watchObservedRunningTime="2026-07-06 19:00:06.784755612 +0000 UTC m=+68.053937958" Jul 06 19:00:06.829314 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:06.828576 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hklfk" podStartSLOduration=67.828556379 podStartE2EDuration="1m7.828556379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:00:06.827156572 +0000 UTC m=+68.096338932" watchObservedRunningTime="2026-07-06 19:00:06.828556379 +0000 UTC m=+68.097738702" Jul 06 19:00:06.851623 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:06.851534 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" podStartSLOduration=66.851510723 podStartE2EDuration="1m6.851510723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:00:06.849703798 +0000 UTC m=+68.118886168" watchObservedRunningTime="2026-07-06 19:00:06.851510723 +0000 UTC m=+68.120693066" Jul 06 19:00:07.766816 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:07.766773 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jmvjq" event={"ID":"e7b6d1cb-2542-4861-9475-25365c8a5d15","Type":"ContainerStarted","Data":"1dedda99c15bfff923826d6c45a218292ecf5c2806b910c08f1c935974e4c3f6"} Jul 06 19:00:07.767235 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:07.766869 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jmvjq" event={"ID":"e7b6d1cb-2542-4861-9475-25365c8a5d15","Type":"ContainerStarted","Data":"6847ef3b7bed5ceadaddf1891490b378c8a899b789a06c11e18c9f07edfddf8f"} Jul 06 19:00:07.767235 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:07.767024 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 19:00:07.768521 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:07.768498 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7456bbb6dc-zfg4t" Jul 06 19:00:07.807249 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:07.807187 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jmvjq" podStartSLOduration=67.177397485 podStartE2EDuration="1m8.807166071s" totalImagesPullingTime="1.629768586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:00:07.786773512 +0000 UTC m=+69.055955855" watchObservedRunningTime="2026-07-06 19:00:07.807166071 +0000 UTC m=+69.076348409" Jul 06 19:00:08.277808 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:08.277778 2553 scope.go:117] "RemoveContainer" containerID="5d4e7d3dd010f87fcf8f9b065746ec7f71359065f25848d42e6b04558b3b9b58" Jul 06 19:00:11.833964 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:11.833929 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94"] Jul 06 19:00:11.871303 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:11.871272 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94"] Jul 06 19:00:11.871489 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:11.871401 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:11.876783 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:11.876750 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 06 19:00:11.876907 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:11.876750 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 19:00:11.877594 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:11.877545 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 06 19:00:11.877594 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:11.877567 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 06 19:00:11.879732 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:11.879216 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 19:00:11.879732 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:11.879258 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 19:00:11.879732 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:11.879460 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 06 19:00:11.960385 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:11.960311 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9kmf\" (UniqueName: \"kubernetes.io/projected/d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1-kube-api-access-l9kmf\") pod \"cluster-proxy-proxy-agent-7cc7c866b4-xcn94\" (UID: \"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:11.960385 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:11.960360 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7cc7c866b4-xcn94\" (UID: \"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:11.960615 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:11.960467 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1-hub\") pod \"cluster-proxy-proxy-agent-7cc7c866b4-xcn94\" (UID: \"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:11.960615 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:11.960515 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1-ca\") pod \"cluster-proxy-proxy-agent-7cc7c866b4-xcn94\" (UID: \"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:11.960615 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:11.960584 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7cc7c866b4-xcn94\" (UID: \"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:11.960734 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:11.960655 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7cc7c866b4-xcn94\" (UID: \"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:12.061375 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:12.061338 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7cc7c866b4-xcn94\" (UID: \"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:12.061375 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:12.061386 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7cc7c866b4-xcn94\" (UID: \"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:12.061617 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:12.061447 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9kmf\" (UniqueName: \"kubernetes.io/projected/d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1-kube-api-access-l9kmf\") pod \"cluster-proxy-proxy-agent-7cc7c866b4-xcn94\" (UID: \"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:12.061617 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:12.061485 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7cc7c866b4-xcn94\" (UID: \"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:12.061617 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:12.061518 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1-hub\") pod \"cluster-proxy-proxy-agent-7cc7c866b4-xcn94\" (UID: \"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:12.061617 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:12.061546 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1-ca\") pod \"cluster-proxy-proxy-agent-7cc7c866b4-xcn94\" (UID: \"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:12.062201 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:12.062176 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7cc7c866b4-xcn94\" (UID: \"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:12.064536 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:12.064492 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1-ca\") pod \"cluster-proxy-proxy-agent-7cc7c866b4-xcn94\" (UID: \"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:12.064629 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:12.064573 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7cc7c866b4-xcn94\" (UID: \"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:12.064766 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:12.064748 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7cc7c866b4-xcn94\" (UID: \"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:12.064919 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:12.064895 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1-hub\") pod \"cluster-proxy-proxy-agent-7cc7c866b4-xcn94\" (UID: \"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:12.078440 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:12.078391 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9kmf\" (UniqueName: \"kubernetes.io/projected/d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1-kube-api-access-l9kmf\") pod \"cluster-proxy-proxy-agent-7cc7c866b4-xcn94\" (UID: \"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:12.194846 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:12.194812 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" Jul 06 19:00:12.902677 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:12.902574 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94"] Jul 06 19:00:12.910345 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:00:12.910317 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2f22b22_dd3d_46f8_9ec1_2e4cccc972f1.slice/crio-7ada311631e1e0cb70aa71a275a494ebbb799ed62ddacf1ae97d6fcad3c41245 WatchSource:0}: Error finding container 7ada311631e1e0cb70aa71a275a494ebbb799ed62ddacf1ae97d6fcad3c41245: Status 404 returned error can't find the container with id 7ada311631e1e0cb70aa71a275a494ebbb799ed62ddacf1ae97d6fcad3c41245 Jul 06 19:00:13.796611 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:13.796580 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:00:13.796803 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:13.796697 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" event={"ID":"1923cd02-24e0-4753-be72-cb22fbd5c05f","Type":"ContainerStarted","Data":"c96a93e4bb1f425f6bdf8778d8f493096ff939fb424f9deaf0f1cedc9e00829c"} Jul 06 19:00:13.797051 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:13.797028 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" Jul 06 19:00:13.799901 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:13.799872 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s9tch" event={"ID":"b345cada-d7a4-4b8f-af05-29d6b7f91edc","Type":"ContainerStarted","Data":"bec347cf919c1c2ea8c5188b9c833e377a71ec6dc0f47d4cc693b5e38cbc76db"} Jul 06 19:00:13.801893 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:13.801872 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mkqz8" event={"ID":"41eea561-3913-46d4-beca-714c42bc07cd","Type":"ContainerStarted","Data":"495368f9f8b3240d3d6c36a0aae29681d9dfcf3854ee69dc3ef0d9c2d75ec9eb"} Jul 06 19:00:13.801983 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:13.801915 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mkqz8" event={"ID":"41eea561-3913-46d4-beca-714c42bc07cd","Type":"ContainerStarted","Data":"376843cd802636c2d1b184554188afddbfa07669706eab40e33cb89085aa648d"} Jul 06 19:00:13.801983 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:13.801957 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mkqz8" Jul 06 19:00:13.803294 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:13.803276 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" Jul 06 19:00:13.803612 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:13.803588 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v5xs7" event={"ID":"f7d2aeda-aa6f-4865-ade0-f9aa24ea785a","Type":"ContainerStarted","Data":"eb13a58c22edc1b3cd27e8073fe2e75d7775ea473cf9e775da27d596d061a3a3"} Jul 06 19:00:13.805569 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:13.805547 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn" event={"ID":"89edb5ad-301f-4971-a47a-82f8078b0c21","Type":"ContainerStarted","Data":"a624515275e40d2b626b4e9ba331a05bbd44f6266e2740404121e32c3afb1999"} Jul 06 19:00:13.805664 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:13.805591 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn" event={"ID":"89edb5ad-301f-4971-a47a-82f8078b0c21","Type":"ContainerStarted","Data":"35d783412931075777cc7524838b11a81fd3358761241dc8d2e1ac1ecf9deb2a"} Jul 06 19:00:13.807297 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:13.807268 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" event={"ID":"2caf149a-9d45-4412-8c6f-006d28f81ef1","Type":"ContainerStarted","Data":"898fdaba3a90f53b356a89ac06e3107320ae58f6ea6552185ea27af4e8ec7d31"} Jul 06 19:00:13.808549 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:13.808526 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" event={"ID":"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1","Type":"ContainerStarted","Data":"7ada311631e1e0cb70aa71a275a494ebbb799ed62ddacf1ae97d6fcad3c41245"} Jul 06 19:00:13.865480 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:13.865400 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-8m9wb" podStartSLOduration=45.013377826 podStartE2EDuration="54.865375011s" totalImagesPullingTime="9.851997185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:00:13.863819731 +0000 UTC m=+75.133002090" watchObservedRunningTime="2026-07-06 19:00:13.865375011 +0000 UTC m=+75.134557353" Jul 06 19:00:13.970336 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:13.969313 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s9tch" podStartSLOduration=17.92711872 podStartE2EDuration="28.969292423s" totalImagesPullingTime="11.042173703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:59:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:00:13.967936441 +0000 UTC m=+75.237118782" watchObservedRunningTime="2026-07-06 19:00:13.969292423 +0000 UTC m=+75.238474766" Jul 06 19:00:14.015487 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:14.015109 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-v5xs7" podStartSLOduration=67.990222102 podStartE2EDuration="1m15.015089208s" totalImagesPullingTime="7.024867106s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 18:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:00:14.01340324 +0000 UTC m=+75.282585583" watchObservedRunningTime="2026-07-06 19:00:14.015089208 +0000 UTC m=+75.284271553" Jul 06 19:00:14.052276 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:14.052172 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-s259v" podStartSLOduration=48.288165928 podStartE2EDuration="55.052151257s" totalImagesPullingTime="6.763985329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:00:14.051300506 +0000 UTC m=+75.320482850" watchObservedRunningTime="2026-07-06 19:00:14.052151257 +0000 UTC m=+75.321333602" Jul 06 19:00:14.103798 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:14.103303 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-rzpjn" podStartSLOduration=48.406408635 podStartE2EDuration="55.103285487s" totalImagesPullingTime="6.696876852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:00:14.102307165 +0000 UTC m=+75.371489507" watchObservedRunningTime="2026-07-06 19:00:14.103285487 +0000 UTC m=+75.372467830" Jul 06 19:00:14.148104 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:14.148044 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mkqz8" podStartSLOduration=34.461409319 podStartE2EDuration="41.148025599s" totalImagesPullingTime="6.68661628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 18:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:00:14.144380962 +0000 UTC m=+75.413563307" watchObservedRunningTime="2026-07-06 19:00:14.148025599 +0000 UTC m=+75.417208133" Jul 06 19:00:17.828277 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:17.828234 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" event={"ID":"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1","Type":"ContainerStarted","Data":"d3b8ebae0bc86d216a5a51b90dd6bbad9764e6bd4febc2f8298bf4ae1d76c606"} Jul 06 19:00:17.828277 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:17.828278 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" event={"ID":"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1","Type":"ContainerStarted","Data":"b0ca55379fe9690a2b197aea6af62cc892282d2d3664443d7f8295e217dbc565"} Jul 06 19:00:17.828743 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:17.828291 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" event={"ID":"d2f22b22-dd3d-46f8-9ec1-2e4cccc972f1","Type":"ContainerStarted","Data":"bbd05959bf86ad22b6210c493def3d91808c11e7c4c756ac575d025eb8118d41"} Jul 06 19:00:17.856248 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:17.856193 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cc7c866b4-xcn94" podStartSLOduration=2.336579707 podStartE2EDuration="6.856178683s" totalImagesPullingTime="4.519598976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:00:17.855698867 +0000 UTC m=+79.124881208" watchObservedRunningTime="2026-07-06 19:00:17.856178683 +0000 UTC m=+79.125361260" Jul 06 19:00:23.817307 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:23.817199 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mkqz8" Jul 06 19:00:25.733360 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:25.733319 2553 patch_prober.go:28] interesting pod/image-registry-57b6b7754b-gx4l2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 19:00:25.733811 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:25.733371 2553 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" podUID="5c9d5b1a-9817-4d71-a7b4-ff6ced34939c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:00:27.647143 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.647110 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fcdhv"] Jul 06 19:00:27.650104 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.650079 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.653079 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.653060 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 19:00:27.653387 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.653371 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 19:00:27.653502 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.653397 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 19:00:27.654181 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.654163 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 19:00:27.672194 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.672173 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cr5b7\"" Jul 06 19:00:27.687801 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.687778 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/838f35e0-7210-421e-9dc1-3130c82ac0c0-sys\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.687894 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.687813 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/838f35e0-7210-421e-9dc1-3130c82ac0c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.687894 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.687834 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/838f35e0-7210-421e-9dc1-3130c82ac0c0-node-exporter-textfile\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.687894 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.687853 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/838f35e0-7210-421e-9dc1-3130c82ac0c0-root\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.687987 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.687898 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/838f35e0-7210-421e-9dc1-3130c82ac0c0-node-exporter-tls\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.687987 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.687958 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/838f35e0-7210-421e-9dc1-3130c82ac0c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.687987 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.687977 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc2n5\" (UniqueName: \"kubernetes.io/projected/838f35e0-7210-421e-9dc1-3130c82ac0c0-kube-api-access-pc2n5\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.688079 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.688004 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/838f35e0-7210-421e-9dc1-3130c82ac0c0-metrics-client-ca\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.688079 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.688030 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/838f35e0-7210-421e-9dc1-3130c82ac0c0-node-exporter-wtmp\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.771610 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.771586 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 19:00:27.788700 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.788673 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/838f35e0-7210-421e-9dc1-3130c82ac0c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.788700 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.788699 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/838f35e0-7210-421e-9dc1-3130c82ac0c0-node-exporter-textfile\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.788883 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.788720 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/838f35e0-7210-421e-9dc1-3130c82ac0c0-root\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.788883 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.788821 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/838f35e0-7210-421e-9dc1-3130c82ac0c0-root\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.788987 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.788877 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/838f35e0-7210-421e-9dc1-3130c82ac0c0-node-exporter-tls\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.788987 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.788972 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/838f35e0-7210-421e-9dc1-3130c82ac0c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.789090 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.789002 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc2n5\" (UniqueName: \"kubernetes.io/projected/838f35e0-7210-421e-9dc1-3130c82ac0c0-kube-api-access-pc2n5\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.789090 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.789052 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/838f35e0-7210-421e-9dc1-3130c82ac0c0-metrics-client-ca\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.789186 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.789090 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/838f35e0-7210-421e-9dc1-3130c82ac0c0-node-exporter-wtmp\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.789186 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.789098 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/838f35e0-7210-421e-9dc1-3130c82ac0c0-node-exporter-textfile\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.789186 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.789172 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/838f35e0-7210-421e-9dc1-3130c82ac0c0-sys\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.789589 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.789396 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/838f35e0-7210-421e-9dc1-3130c82ac0c0-sys\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.789589 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.789445 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/838f35e0-7210-421e-9dc1-3130c82ac0c0-node-exporter-wtmp\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.789589 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.789538 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/838f35e0-7210-421e-9dc1-3130c82ac0c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.789800 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.789689 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/838f35e0-7210-421e-9dc1-3130c82ac0c0-metrics-client-ca\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.791245 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.791225 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/838f35e0-7210-421e-9dc1-3130c82ac0c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.791390 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.791335 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/838f35e0-7210-421e-9dc1-3130c82ac0c0-node-exporter-tls\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.805371 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.805345 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc2n5\" (UniqueName: \"kubernetes.io/projected/838f35e0-7210-421e-9dc1-3130c82ac0c0-kube-api-access-pc2n5\") pod \"node-exporter-fcdhv\" (UID: \"838f35e0-7210-421e-9dc1-3130c82ac0c0\") " pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.958973 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:27.958888 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fcdhv" Jul 06 19:00:27.969024 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:00:27.968994 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838f35e0_7210_421e_9dc1_3130c82ac0c0.slice/crio-cd1f4d1824951271f647472c49b7c381d635ef4502dcc52b2d10dd60ff59cc0c WatchSource:0}: Error finding container cd1f4d1824951271f647472c49b7c381d635ef4502dcc52b2d10dd60ff59cc0c: Status 404 returned error can't find the container with id cd1f4d1824951271f647472c49b7c381d635ef4502dcc52b2d10dd60ff59cc0c Jul 06 19:00:28.868975 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:28.868942 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcdhv" event={"ID":"838f35e0-7210-421e-9dc1-3130c82ac0c0","Type":"ContainerStarted","Data":"cd1f4d1824951271f647472c49b7c381d635ef4502dcc52b2d10dd60ff59cc0c"} Jul 06 19:00:29.874232 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:29.874198 2553 generic.go:356] "Generic (PLEG): container finished" podID="838f35e0-7210-421e-9dc1-3130c82ac0c0" containerID="a334a3f55ebef56ec50721c7efcac2094bbc4166c3b95b3c22796461302924cf" exitCode=0 Jul 06 19:00:29.874623 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:29.874239 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcdhv" event={"ID":"838f35e0-7210-421e-9dc1-3130c82ac0c0","Type":"ContainerDied","Data":"a334a3f55ebef56ec50721c7efcac2094bbc4166c3b95b3c22796461302924cf"} Jul 06 19:00:30.880269 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:30.880223 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcdhv" event={"ID":"838f35e0-7210-421e-9dc1-3130c82ac0c0","Type":"ContainerStarted","Data":"267d3ee3e700b0ac62c9ae530be86182dffdd030542f8693f81fb253acc91214"} Jul 06 19:00:30.880665 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:30.880294 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcdhv" event={"ID":"838f35e0-7210-421e-9dc1-3130c82ac0c0","Type":"ContainerStarted","Data":"fd681b0f358bd18d7e60b74772788e7a8dd6e657d409fb6f17a9c3312d056a50"} Jul 06 19:00:30.907654 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:30.907602 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fcdhv" podStartSLOduration=2.7243201360000002 podStartE2EDuration="3.90758394s" totalImagesPullingTime="1.107660618s" totalInitContainerRuntime="75.603186ms" isStatefulPod=true podCreationTimestamp="2026-07-06 19:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:00:30.907034133 +0000 UTC m=+92.176216474" watchObservedRunningTime="2026-07-06 19:00:30.90758394 +0000 UTC m=+92.176766285" Jul 06 19:00:34.416204 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:34.416170 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57b6b7754b-gx4l2"] Jul 06 19:00:37.770168 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:37.770140 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hklfk" Jul 06 19:00:59.436523 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.436475 2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" podUID="5c9d5b1a-9817-4d71-a7b4-ff6ced34939c" containerName="registry" containerID="cri-o://520d6e83f8246754eb9a4aaf6343acda1175b8a5a2d1f0a30668107869bd5684" gracePeriod=30 Jul 06 19:00:59.476317 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:00:59.476289 2553 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c9d5b1a_9817_4d71_a7b4_ff6ced34939c.slice/crio-conmon-520d6e83f8246754eb9a4aaf6343acda1175b8a5a2d1f0a30668107869bd5684.scope\": RecentStats: unable to find data in memory cache]" Jul 06 19:00:59.682557 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.682535 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 19:00:59.749690 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.749660 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-certificates\") pod \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " Jul 06 19:00:59.749866 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.749711 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls\") pod \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " Jul 06 19:00:59.749866 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.749742 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r92tv\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-kube-api-access-r92tv\") pod \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " Jul 06 19:00:59.749866 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.749757 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-trusted-ca\") pod \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " Jul 06 19:00:59.749866 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.749790 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-image-registry-private-configuration\") pod \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " Jul 06 19:00:59.749866 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.749821 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-ca-trust-extracted\") pod \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " Jul 06 19:00:59.749866 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.749864 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-bound-sa-token\") pod \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " Jul 06 19:00:59.750175 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.749898 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-installation-pull-secrets\") pod \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\" (UID: \"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c\") " Jul 06 19:00:59.750229 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.750167 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c" (UID: "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:00:59.750386 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.750346 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c" (UID: "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:00:59.752569 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.752543 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c" (UID: "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:00:59.752677 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.752615 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c" (UID: "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:00:59.752765 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.752733 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c" (UID: "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:00:59.752813 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.752793 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c" (UID: "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:00:59.752910 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.752880 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-kube-api-access-r92tv" (OuterVolumeSpecName: "kube-api-access-r92tv") pod "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c" (UID: "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c"). InnerVolumeSpecName "kube-api-access-r92tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:00:59.759175 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.759151 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c" (UID: "5c9d5b1a-9817-4d71-a7b4-ff6ced34939c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:00:59.850978 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.850948 2553 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-bound-sa-token\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:00:59.850978 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.850972 2553 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-installation-pull-secrets\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:00:59.850978 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.850983 2553 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-certificates\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:00:59.851180 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.850992 2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-registry-tls\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:00:59.851180 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.851001 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r92tv\" (UniqueName: \"kubernetes.io/projected/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-kube-api-access-r92tv\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:00:59.851180 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.851009 2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-trusted-ca\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:00:59.851180 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.851019 2553 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-image-registry-private-configuration\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:00:59.851180 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.851028 2553 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c-ca-trust-extracted\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:00:59.981646 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.981618 2553 generic.go:356] "Generic (PLEG): container finished" podID="5c9d5b1a-9817-4d71-a7b4-ff6ced34939c" containerID="520d6e83f8246754eb9a4aaf6343acda1175b8a5a2d1f0a30668107869bd5684" exitCode=0 Jul 06 19:00:59.981791 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.981660 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" Jul 06 19:00:59.981791 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.981685 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" event={"ID":"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c","Type":"ContainerDied","Data":"520d6e83f8246754eb9a4aaf6343acda1175b8a5a2d1f0a30668107869bd5684"} Jul 06 19:00:59.981791 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.981724 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57b6b7754b-gx4l2" event={"ID":"5c9d5b1a-9817-4d71-a7b4-ff6ced34939c","Type":"ContainerDied","Data":"c8a25ac40a87f10a1eef4cebf1240c4ab64ae30df76a5678147a258f8dc6fda0"} Jul 06 19:00:59.981791 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.981742 2553 scope.go:117] "RemoveContainer" containerID="520d6e83f8246754eb9a4aaf6343acda1175b8a5a2d1f0a30668107869bd5684" Jul 06 19:00:59.991984 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.991963 2553 scope.go:117] "RemoveContainer" containerID="520d6e83f8246754eb9a4aaf6343acda1175b8a5a2d1f0a30668107869bd5684" Jul 06 19:00:59.992249 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:00:59.992230 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520d6e83f8246754eb9a4aaf6343acda1175b8a5a2d1f0a30668107869bd5684\": container with ID starting with 520d6e83f8246754eb9a4aaf6343acda1175b8a5a2d1f0a30668107869bd5684 not found: ID does not exist" containerID="520d6e83f8246754eb9a4aaf6343acda1175b8a5a2d1f0a30668107869bd5684" Jul 06 19:00:59.992321 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:00:59.992255 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520d6e83f8246754eb9a4aaf6343acda1175b8a5a2d1f0a30668107869bd5684"} err="failed to get container status \"520d6e83f8246754eb9a4aaf6343acda1175b8a5a2d1f0a30668107869bd5684\": rpc error: code = NotFound desc = could not find container \"520d6e83f8246754eb9a4aaf6343acda1175b8a5a2d1f0a30668107869bd5684\": container with ID starting with 520d6e83f8246754eb9a4aaf6343acda1175b8a5a2d1f0a30668107869bd5684 not found: ID does not exist" Jul 06 19:01:00.003549 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:01:00.003495 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57b6b7754b-gx4l2"] Jul 06 19:01:00.014013 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:01:00.013989 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57b6b7754b-gx4l2"] Jul 06 19:01:01.281703 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:01:01.281673 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c9d5b1a-9817-4d71-a7b4-ff6ced34939c" path="/var/lib/kubelet/pods/5c9d5b1a-9817-4d71-a7b4-ff6ced34939c/volumes" Jul 06 19:01:06.005678 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:01:06.005643 2553 generic.go:356] "Generic (PLEG): container finished" podID="628f818d-2286-4af1-b946-5779244f0bff" containerID="4c2f856d91adeab397e21a63ba478e97c67a2469597f3a095362fd3e36c0ca74" exitCode=0 Jul 06 19:01:06.006107 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:01:06.005719 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s8v9j" event={"ID":"628f818d-2286-4af1-b946-5779244f0bff","Type":"ContainerDied","Data":"4c2f856d91adeab397e21a63ba478e97c67a2469597f3a095362fd3e36c0ca74"} Jul 06 19:01:06.006107 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:01:06.006044 2553 scope.go:117] "RemoveContainer" containerID="4c2f856d91adeab397e21a63ba478e97c67a2469597f3a095362fd3e36c0ca74" Jul 06 19:01:07.010924 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:01:07.010896 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-s8v9j" event={"ID":"628f818d-2286-4af1-b946-5779244f0bff","Type":"ContainerStarted","Data":"bb1d684da9c9635d73d6ad2615c13765dc75ca7dc158d46108bd81f19e757efb"} Jul 06 19:01:10.021990 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:01:10.021960 2553 generic.go:356] "Generic (PLEG): container finished" podID="724d10df-ec14-4dcd-8b9c-5265acf5ed80" containerID="37185797dcb4b94dc3017011418a3e2fb4a9e37ff196087852b1e2f17af6fd8a" exitCode=0 Jul 06 19:01:10.022371 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:01:10.022036 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" event={"ID":"724d10df-ec14-4dcd-8b9c-5265acf5ed80","Type":"ContainerDied","Data":"37185797dcb4b94dc3017011418a3e2fb4a9e37ff196087852b1e2f17af6fd8a"} Jul 06 19:01:10.022371 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:01:10.022350 2553 scope.go:117] "RemoveContainer" containerID="37185797dcb4b94dc3017011418a3e2fb4a9e37ff196087852b1e2f17af6fd8a" Jul 06 19:01:11.027457 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:01:11.027405 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-tsj58" event={"ID":"724d10df-ec14-4dcd-8b9c-5265acf5ed80","Type":"ContainerStarted","Data":"2a51922b6843a19453b10e2a0a1dc0af8b1c2a402ec7321ca6c12b56a2177e78"} Jul 06 19:01:11.028870 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:01:11.028848 2553 generic.go:356] "Generic (PLEG): container finished" podID="0c10df95-ff62-4f94-a74d-91742dc4c92c" containerID="39ad1bee409b5bc80a166d81241d5854a2ce417976d35c7a18ab5d070d113e67" exitCode=0 Jul 06 19:01:11.028937 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:01:11.028891 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-jjlwh" event={"ID":"0c10df95-ff62-4f94-a74d-91742dc4c92c","Type":"ContainerDied","Data":"39ad1bee409b5bc80a166d81241d5854a2ce417976d35c7a18ab5d070d113e67"} Jul 06 19:01:11.029136 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:01:11.029125 2553 scope.go:117] "RemoveContainer" containerID="39ad1bee409b5bc80a166d81241d5854a2ce417976d35c7a18ab5d070d113e67" Jul 06 19:01:12.034277 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:01:12.034242 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-jjlwh" event={"ID":"0c10df95-ff62-4f94-a74d-91742dc4c92c","Type":"ContainerStarted","Data":"432792aea5716e902fdd361e22f3b9c54b5bd5000bc6ebff0d84248ffa708fe8"} Jul 06 19:03:59.126564 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:03:59.126532 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:03:59.132485 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:03:59.132462 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:03:59.133454 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:03:59.133433 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:03:59.136712 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:03:59.136689 2553 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 19:03:59.140509 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:03:59.140489 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:05:35.839285 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.839248 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w"] Jul 06 19:05:35.839765 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.839544 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c9d5b1a-9817-4d71-a7b4-ff6ced34939c" containerName="registry" Jul 06 19:05:35.839765 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.839556 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c9d5b1a-9817-4d71-a7b4-ff6ced34939c" containerName="registry" Jul 06 19:05:35.839765 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.839635 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c9d5b1a-9817-4d71-a7b4-ff6ced34939c" containerName="registry" Jul 06 19:05:35.842453 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.842432 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" Jul 06 19:05:35.849106 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.849086 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 19:05:35.861027 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.861005 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 19:05:35.861605 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.861591 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 19:05:35.861661 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.861605 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 19:05:35.861661 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.861608 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kmtgc\"" Jul 06 19:05:35.890665 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.890635 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w"] Jul 06 19:05:35.896070 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.896049 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9fa46fe-a84d-449b-a63b-47121b01ba82-webhook-cert\") pod \"opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w\" (UID: \"c9fa46fe-a84d-449b-a63b-47121b01ba82\") " pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" Jul 06 19:05:35.896160 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.896090 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c9fa46fe-a84d-449b-a63b-47121b01ba82-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w\" (UID: \"c9fa46fe-a84d-449b-a63b-47121b01ba82\") " pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" Jul 06 19:05:35.896160 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.896121 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9fa46fe-a84d-449b-a63b-47121b01ba82-tmp\") pod \"opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w\" (UID: \"c9fa46fe-a84d-449b-a63b-47121b01ba82\") " pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" Jul 06 19:05:35.896233 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.896175 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c9fa46fe-a84d-449b-a63b-47121b01ba82-manifests\") pod \"opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w\" (UID: \"c9fa46fe-a84d-449b-a63b-47121b01ba82\") " pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" Jul 06 19:05:35.896233 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.896224 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knmnz\" (UniqueName: \"kubernetes.io/projected/c9fa46fe-a84d-449b-a63b-47121b01ba82-kube-api-access-knmnz\") pod \"opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w\" (UID: \"c9fa46fe-a84d-449b-a63b-47121b01ba82\") " pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" Jul 06 19:05:35.996681 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.996652 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9fa46fe-a84d-449b-a63b-47121b01ba82-tmp\") pod \"opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w\" (UID: \"c9fa46fe-a84d-449b-a63b-47121b01ba82\") " pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" Jul 06 19:05:35.996858 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.996688 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c9fa46fe-a84d-449b-a63b-47121b01ba82-manifests\") pod \"opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w\" (UID: \"c9fa46fe-a84d-449b-a63b-47121b01ba82\") " pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" Jul 06 19:05:35.996858 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.996710 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knmnz\" (UniqueName: \"kubernetes.io/projected/c9fa46fe-a84d-449b-a63b-47121b01ba82-kube-api-access-knmnz\") pod \"opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w\" (UID: \"c9fa46fe-a84d-449b-a63b-47121b01ba82\") " pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" Jul 06 19:05:35.996858 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.996734 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9fa46fe-a84d-449b-a63b-47121b01ba82-webhook-cert\") pod \"opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w\" (UID: \"c9fa46fe-a84d-449b-a63b-47121b01ba82\") " pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" Jul 06 19:05:35.996858 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.996772 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c9fa46fe-a84d-449b-a63b-47121b01ba82-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w\" (UID: \"c9fa46fe-a84d-449b-a63b-47121b01ba82\") " pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" Jul 06 19:05:35.997139 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.997114 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c9fa46fe-a84d-449b-a63b-47121b01ba82-manifests\") pod \"opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w\" (UID: \"c9fa46fe-a84d-449b-a63b-47121b01ba82\") " pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" Jul 06 19:05:35.999144 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.999113 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9fa46fe-a84d-449b-a63b-47121b01ba82-tmp\") pod \"opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w\" (UID: \"c9fa46fe-a84d-449b-a63b-47121b01ba82\") " pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" Jul 06 19:05:35.999362 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.999340 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c9fa46fe-a84d-449b-a63b-47121b01ba82-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w\" (UID: \"c9fa46fe-a84d-449b-a63b-47121b01ba82\") " pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" Jul 06 19:05:35.999464 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:35.999363 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c9fa46fe-a84d-449b-a63b-47121b01ba82-webhook-cert\") pod \"opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w\" (UID: \"c9fa46fe-a84d-449b-a63b-47121b01ba82\") " pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" Jul 06 19:05:36.012359 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:36.012339 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knmnz\" (UniqueName: \"kubernetes.io/projected/c9fa46fe-a84d-449b-a63b-47121b01ba82-kube-api-access-knmnz\") pod \"opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w\" (UID: \"c9fa46fe-a84d-449b-a63b-47121b01ba82\") " pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" Jul 06 19:05:36.151892 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:36.151816 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" Jul 06 19:05:36.300150 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:36.300127 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w"] Jul 06 19:05:36.302483 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:05:36.302451 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9fa46fe_a84d_449b_a63b_47121b01ba82.slice/crio-b3a9861a6f42caf4f0c717712b9da2a85e1ba3a61b09d0ad4d429d01b2ba9f06 WatchSource:0}: Error finding container b3a9861a6f42caf4f0c717712b9da2a85e1ba3a61b09d0ad4d429d01b2ba9f06: Status 404 returned error can't find the container with id b3a9861a6f42caf4f0c717712b9da2a85e1ba3a61b09d0ad4d429d01b2ba9f06 Jul 06 19:05:36.304174 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:36.304153 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:05:36.974017 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:36.973910 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" event={"ID":"c9fa46fe-a84d-449b-a63b-47121b01ba82","Type":"ContainerStarted","Data":"b3a9861a6f42caf4f0c717712b9da2a85e1ba3a61b09d0ad4d429d01b2ba9f06"} Jul 06 19:05:39.992170 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:39.992137 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" event={"ID":"c9fa46fe-a84d-449b-a63b-47121b01ba82","Type":"ContainerStarted","Data":"508831c18e36377e9a9c6f55781074dc665d3c53d62eb136242d88c5f5462205"} Jul 06 19:05:40.997523 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:40.997484 2553 generic.go:356] "Generic (PLEG): container finished" podID="c9fa46fe-a84d-449b-a63b-47121b01ba82" containerID="508831c18e36377e9a9c6f55781074dc665d3c53d62eb136242d88c5f5462205" exitCode=0 Jul 06 19:05:40.997895 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:40.997530 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" event={"ID":"c9fa46fe-a84d-449b-a63b-47121b01ba82","Type":"ContainerDied","Data":"508831c18e36377e9a9c6f55781074dc665d3c53d62eb136242d88c5f5462205"} Jul 06 19:05:42.003686 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:42.003649 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" event={"ID":"c9fa46fe-a84d-449b-a63b-47121b01ba82","Type":"ContainerStarted","Data":"168a817210d084687625200a3772d03c8c543f39a776cb569b255ee8550dd651"} Jul 06 19:05:42.004092 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:42.003789 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" Jul 06 19:05:42.027748 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:42.027704 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" podStartSLOduration=2.725281287 podStartE2EDuration="7.027690112s" totalImagesPullingTime="3.262884895s" totalInitContainerRuntime="1.03952393s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:05:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:05:42.027201938 +0000 UTC m=+403.296384292" watchObservedRunningTime="2026-07-06 19:05:42.027690112 +0000 UTC m=+403.296872433" Jul 06 19:05:53.009915 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:53.009883 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w" Jul 06 19:05:56.580914 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:56.580883 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c9545dd54-sjc64"] Jul 06 19:05:56.587116 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:56.587090 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c9545dd54-sjc64" Jul 06 19:05:56.590910 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:05:56.590871 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kube-auth-proxy-7c9545dd54-sjc64\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="9780aa2b-78ab-489e-be4b-cefc925b9087" pod="openshift-ingress/kube-auth-proxy-7c9545dd54-sjc64" Jul 06 19:05:56.590910 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:05:56.590887 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-auth-proxy-tls\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret" Jul 06 19:05:56.590910 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:05:56.590900 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-auth-proxy-creds\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" type="*v1.Secret" Jul 06 19:05:56.651711 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:56.651684 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9780aa2b-78ab-489e-be4b-cefc925b9087-tls-certs\") pod \"kube-auth-proxy-7c9545dd54-sjc64\" (UID: \"9780aa2b-78ab-489e-be4b-cefc925b9087\") " pod="openshift-ingress/kube-auth-proxy-7c9545dd54-sjc64" Jul 06 19:05:56.651872 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:56.651728 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9780aa2b-78ab-489e-be4b-cefc925b9087-tmp\") pod \"kube-auth-proxy-7c9545dd54-sjc64\" (UID: \"9780aa2b-78ab-489e-be4b-cefc925b9087\") " pod="openshift-ingress/kube-auth-proxy-7c9545dd54-sjc64" Jul 06 19:05:56.651872 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:56.651747 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmxkn\" (UniqueName: \"kubernetes.io/projected/9780aa2b-78ab-489e-be4b-cefc925b9087-kube-api-access-tmxkn\") pod \"kube-auth-proxy-7c9545dd54-sjc64\" (UID: \"9780aa2b-78ab-489e-be4b-cefc925b9087\") " pod="openshift-ingress/kube-auth-proxy-7c9545dd54-sjc64" Jul 06 19:05:56.753056 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:56.753026 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9780aa2b-78ab-489e-be4b-cefc925b9087-tls-certs\") pod \"kube-auth-proxy-7c9545dd54-sjc64\" (UID: \"9780aa2b-78ab-489e-be4b-cefc925b9087\") " pod="openshift-ingress/kube-auth-proxy-7c9545dd54-sjc64" Jul 06 19:05:56.753213 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:56.753081 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9780aa2b-78ab-489e-be4b-cefc925b9087-tmp\") pod \"kube-auth-proxy-7c9545dd54-sjc64\" (UID: \"9780aa2b-78ab-489e-be4b-cefc925b9087\") " pod="openshift-ingress/kube-auth-proxy-7c9545dd54-sjc64" Jul 06 19:05:56.753213 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:56.753105 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmxkn\" (UniqueName: \"kubernetes.io/projected/9780aa2b-78ab-489e-be4b-cefc925b9087-kube-api-access-tmxkn\") pod \"kube-auth-proxy-7c9545dd54-sjc64\" (UID: \"9780aa2b-78ab-489e-be4b-cefc925b9087\") " pod="openshift-ingress/kube-auth-proxy-7c9545dd54-sjc64" Jul 06 19:05:56.755482 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:56.755459 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9780aa2b-78ab-489e-be4b-cefc925b9087-tmp\") pod \"kube-auth-proxy-7c9545dd54-sjc64\" (UID: \"9780aa2b-78ab-489e-be4b-cefc925b9087\") " pod="openshift-ingress/kube-auth-proxy-7c9545dd54-sjc64" Jul 06 19:05:56.783541 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:56.783514 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c9545dd54-sjc64"] Jul 06 19:05:56.856759 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:56.856689 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmxkn\" (UniqueName: \"kubernetes.io/projected/9780aa2b-78ab-489e-be4b-cefc925b9087-kube-api-access-tmxkn\") pod \"kube-auth-proxy-7c9545dd54-sjc64\" (UID: \"9780aa2b-78ab-489e-be4b-cefc925b9087\") " pod="openshift-ingress/kube-auth-proxy-7c9545dd54-sjc64" Jul 06 19:05:57.405803 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:57.405775 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 19:05:57.754143 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:05:57.754106 2553 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: failed to sync secret cache: timed out waiting for the condition Jul 06 19:05:57.754541 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:05:57.754214 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9780aa2b-78ab-489e-be4b-cefc925b9087-tls-certs podName:9780aa2b-78ab-489e-be4b-cefc925b9087 nodeName:}" failed. No retries permitted until 2026-07-06 19:05:58.254195503 +0000 UTC m=+419.523377827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9780aa2b-78ab-489e-be4b-cefc925b9087-tls-certs") pod "kube-auth-proxy-7c9545dd54-sjc64" (UID: "9780aa2b-78ab-489e-be4b-cefc925b9087") : failed to sync secret cache: timed out waiting for the condition Jul 06 19:05:58.039537 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:58.039451 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 19:05:58.265352 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:58.265321 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9780aa2b-78ab-489e-be4b-cefc925b9087-tls-certs\") pod \"kube-auth-proxy-7c9545dd54-sjc64\" (UID: \"9780aa2b-78ab-489e-be4b-cefc925b9087\") " pod="openshift-ingress/kube-auth-proxy-7c9545dd54-sjc64" Jul 06 19:05:58.267821 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:58.267798 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9780aa2b-78ab-489e-be4b-cefc925b9087-tls-certs\") pod \"kube-auth-proxy-7c9545dd54-sjc64\" (UID: \"9780aa2b-78ab-489e-be4b-cefc925b9087\") " pod="openshift-ingress/kube-auth-proxy-7c9545dd54-sjc64" Jul 06 19:05:58.367531 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:58.367461 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5psq2"] Jul 06 19:05:58.369701 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:58.369687 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-5psq2" Jul 06 19:05:58.372637 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:58.372614 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 06 19:05:58.396123 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:58.396097 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c9545dd54-sjc64" Jul 06 19:05:58.433043 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:58.433015 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5psq2"] Jul 06 19:05:58.466917 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:58.466886 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hvw5\" (UniqueName: \"kubernetes.io/projected/d59bbf4a-f17c-43a9-ab25-d64b4b351819-kube-api-access-9hvw5\") pod \"model-serving-api-54d89bd5fc-5psq2\" (UID: \"d59bbf4a-f17c-43a9-ab25-d64b4b351819\") " pod="opendatahub/model-serving-api-54d89bd5fc-5psq2" Jul 06 19:05:58.467126 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:58.467103 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59bbf4a-f17c-43a9-ab25-d64b4b351819-tls-certs\") pod \"model-serving-api-54d89bd5fc-5psq2\" (UID: \"d59bbf4a-f17c-43a9-ab25-d64b4b351819\") " pod="opendatahub/model-serving-api-54d89bd5fc-5psq2" Jul 06 19:05:58.544136 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:58.544103 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c9545dd54-sjc64"] Jul 06 19:05:58.547281 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:05:58.547253 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9780aa2b_78ab_489e_be4b_cefc925b9087.slice/crio-47b40aab87f324910f9c057c36ac45592ba3a4fa976404013e0fb1f6dbebaffe WatchSource:0}: Error finding container 47b40aab87f324910f9c057c36ac45592ba3a4fa976404013e0fb1f6dbebaffe: Status 404 returned error can't find the container with id 47b40aab87f324910f9c057c36ac45592ba3a4fa976404013e0fb1f6dbebaffe Jul 06 19:05:58.568355 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:58.568332 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hvw5\" (UniqueName: \"kubernetes.io/projected/d59bbf4a-f17c-43a9-ab25-d64b4b351819-kube-api-access-9hvw5\") pod \"model-serving-api-54d89bd5fc-5psq2\" (UID: \"d59bbf4a-f17c-43a9-ab25-d64b4b351819\") " pod="opendatahub/model-serving-api-54d89bd5fc-5psq2" Jul 06 19:05:58.568491 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:58.568376 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59bbf4a-f17c-43a9-ab25-d64b4b351819-tls-certs\") pod \"model-serving-api-54d89bd5fc-5psq2\" (UID: \"d59bbf4a-f17c-43a9-ab25-d64b4b351819\") " pod="opendatahub/model-serving-api-54d89bd5fc-5psq2" Jul 06 19:05:58.568611 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:05:58.568591 2553 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 06 19:05:58.568674 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:05:58.568664 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d59bbf4a-f17c-43a9-ab25-d64b4b351819-tls-certs podName:d59bbf4a-f17c-43a9-ab25-d64b4b351819 nodeName:}" failed. No retries permitted until 2026-07-06 19:05:59.068643091 +0000 UTC m=+420.337825431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/d59bbf4a-f17c-43a9-ab25-d64b4b351819-tls-certs") pod "model-serving-api-54d89bd5fc-5psq2" (UID: "d59bbf4a-f17c-43a9-ab25-d64b4b351819") : secret "model-serving-api-tls" not found Jul 06 19:05:58.583953 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:58.583933 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hvw5\" (UniqueName: \"kubernetes.io/projected/d59bbf4a-f17c-43a9-ab25-d64b4b351819-kube-api-access-9hvw5\") pod \"model-serving-api-54d89bd5fc-5psq2\" (UID: \"d59bbf4a-f17c-43a9-ab25-d64b4b351819\") " pod="opendatahub/model-serving-api-54d89bd5fc-5psq2" Jul 06 19:05:59.064790 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:59.064744 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c9545dd54-sjc64" event={"ID":"9780aa2b-78ab-489e-be4b-cefc925b9087","Type":"ContainerStarted","Data":"47b40aab87f324910f9c057c36ac45592ba3a4fa976404013e0fb1f6dbebaffe"} Jul 06 19:05:59.072245 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:59.072213 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59bbf4a-f17c-43a9-ab25-d64b4b351819-tls-certs\") pod \"model-serving-api-54d89bd5fc-5psq2\" (UID: \"d59bbf4a-f17c-43a9-ab25-d64b4b351819\") " pod="opendatahub/model-serving-api-54d89bd5fc-5psq2" Jul 06 19:05:59.075100 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:59.075076 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59bbf4a-f17c-43a9-ab25-d64b4b351819-tls-certs\") pod \"model-serving-api-54d89bd5fc-5psq2\" (UID: \"d59bbf4a-f17c-43a9-ab25-d64b4b351819\") " pod="opendatahub/model-serving-api-54d89bd5fc-5psq2" Jul 06 19:05:59.279491 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:59.279336 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-5psq2" Jul 06 19:05:59.481448 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:05:59.481402 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-5psq2"] Jul 06 19:05:59.485921 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:05:59.485891 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59bbf4a_f17c_43a9_ab25_d64b4b351819.slice/crio-80ced261f2750a5db0d352c65e08efa9c5b1b7cfc89b30db7c48b8da8fb3e44d WatchSource:0}: Error finding container 80ced261f2750a5db0d352c65e08efa9c5b1b7cfc89b30db7c48b8da8fb3e44d: Status 404 returned error can't find the container with id 80ced261f2750a5db0d352c65e08efa9c5b1b7cfc89b30db7c48b8da8fb3e44d Jul 06 19:06:00.072553 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.072514 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-5psq2" event={"ID":"d59bbf4a-f17c-43a9-ab25-d64b4b351819","Type":"ContainerStarted","Data":"80ced261f2750a5db0d352c65e08efa9c5b1b7cfc89b30db7c48b8da8fb3e44d"} Jul 06 19:06:00.349256 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.349175 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5qwft"] Jul 06 19:06:00.351605 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.351584 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.366655 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.366634 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 06 19:06:00.366931 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.366724 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 06 19:06:00.366931 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.366856 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 06 19:06:00.367148 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.367130 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 19:06:00.367215 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.367133 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xbd6q\"" Jul 06 19:06:00.417150 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.417126 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5qwft"] Jul 06 19:06:00.484741 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.484708 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.484910 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.484748 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-local-certs\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.484910 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.484821 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-token\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.484910 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.484843 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.485065 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.484922 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-cacerts\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.485065 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.484962 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.485065 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.484983 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhktc\" (UniqueName: \"kubernetes.io/projected/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-kube-api-access-lhktc\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.585393 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.585359 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhktc\" (UniqueName: \"kubernetes.io/projected/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-kube-api-access-lhktc\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.585587 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.585442 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.585587 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.585474 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-local-certs\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.585587 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.585519 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-token\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.585587 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.585547 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.585820 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.585593 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-cacerts\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.585820 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.585628 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.586631 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.586582 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.588479 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.588451 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.588784 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.588764 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-cacerts\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.588867 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.588803 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.589238 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.589214 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-local-certs\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.600377 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.600321 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhktc\" (UniqueName: \"kubernetes.io/projected/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-kube-api-access-lhktc\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.601239 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.601216 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-token\") pod \"istiod-openshift-gateway-7574459656-5qwft\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.664302 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.664276 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:00.839330 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:00.839311 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5qwft"] Jul 06 19:06:01.354004 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:06:01.353970 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0ba8c8_2ff4_491d_bc37_4e68e088f071.slice/crio-c8b0a3db87ddd2fab00c07eadf825f73bded83f661bcca57cc0bca4796ec0138 WatchSource:0}: Error finding container c8b0a3db87ddd2fab00c07eadf825f73bded83f661bcca57cc0bca4796ec0138: Status 404 returned error can't find the container with id c8b0a3db87ddd2fab00c07eadf825f73bded83f661bcca57cc0bca4796ec0138 Jul 06 19:06:01.725625 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:01.725603 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 19:06:02.083447 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:02.083339 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-5psq2" event={"ID":"d59bbf4a-f17c-43a9-ab25-d64b4b351819","Type":"ContainerStarted","Data":"0f52b18cad59d0dc98f8f0e379b26b9fa27b34d9b8ae3f1b9e0b80560070585d"} Jul 06 19:06:02.083611 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:02.083467 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-5psq2" Jul 06 19:06:02.084853 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:02.084821 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" event={"ID":"5e0ba8c8-2ff4-491d-bc37-4e68e088f071","Type":"ContainerStarted","Data":"c8b0a3db87ddd2fab00c07eadf825f73bded83f661bcca57cc0bca4796ec0138"} Jul 06 19:06:02.086614 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:02.086585 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c9545dd54-sjc64" event={"ID":"9780aa2b-78ab-489e-be4b-cefc925b9087","Type":"ContainerStarted","Data":"fa17b73d1fb6c3238473cd6866e30c4df3a877e7a2eabcff7f0c2c41adcb492e"} Jul 06 19:06:02.110439 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:02.110375 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-5psq2" podStartSLOduration=1.877162808 podStartE2EDuration="4.110355415s" totalImagesPullingTime="2.233192607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:02.108989369 +0000 UTC m=+423.378171712" watchObservedRunningTime="2026-07-06 19:06:02.110355415 +0000 UTC m=+423.379537759" Jul 06 19:06:02.134289 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:02.134242 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c9545dd54-sjc64" podStartSLOduration=2.960735712 podStartE2EDuration="6.134226771s" totalImagesPullingTime="3.173491059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:05:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:02.133123753 +0000 UTC m=+423.402306105" watchObservedRunningTime="2026-07-06 19:06:02.134226771 +0000 UTC m=+423.403409114" Jul 06 19:06:03.664026 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:03.663968 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:06:03.664271 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:03.664096 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:06:04.097383 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:04.097344 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" event={"ID":"5e0ba8c8-2ff4-491d-bc37-4e68e088f071","Type":"ContainerStarted","Data":"aa8b5e1b5a3c46becae1b8441ca9ce6d0bd0ca9dac0e6fe3e739ff783abdcaab"} Jul 06 19:06:04.097641 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:04.097619 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:04.099461 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:04.099410 2553 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-5qwft container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 19:06:04.099569 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:04.099481 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" podUID="5e0ba8c8-2ff4-491d-bc37-4e68e088f071" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:06:05.102629 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:05.102593 2553 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-5qwft container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 19:06:05.103066 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:05.102650 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" podUID="5e0ba8c8-2ff4-491d-bc37-4e68e088f071" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 19:06:07.379548 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.379494 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" podStartSLOduration=5.0723253249999996 podStartE2EDuration="7.379471966s" totalImagesPullingTime="2.307146641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:04.153667337 +0000 UTC m=+425.422849679" watchObservedRunningTime="2026-07-06 19:06:07.379471966 +0000 UTC m=+428.648654307" Jul 06 19:06:07.380041 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.379658 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6"] Jul 06 19:06:07.382345 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.382323 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.385104 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.385082 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-ldjs9\"" Jul 06 19:06:07.416511 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.416485 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6"] Jul 06 19:06:07.443452 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.443404 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea08144-1c62-42ec-8d15-3da05e0fa17b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.443570 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.443475 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea08144-1c62-42ec-8d15-3da05e0fa17b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.443570 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.443501 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea08144-1c62-42ec-8d15-3da05e0fa17b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.443570 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.443551 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea08144-1c62-42ec-8d15-3da05e0fa17b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.443669 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.443594 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea08144-1c62-42ec-8d15-3da05e0fa17b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.443669 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.443620 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea08144-1c62-42ec-8d15-3da05e0fa17b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.443669 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.443644 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea08144-1c62-42ec-8d15-3da05e0fa17b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.443669 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.443662 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea08144-1c62-42ec-8d15-3da05e0fa17b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.443791 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.443678 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phpp9\" (UniqueName: \"kubernetes.io/projected/6ea08144-1c62-42ec-8d15-3da05e0fa17b-kube-api-access-phpp9\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.544866 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.544830 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea08144-1c62-42ec-8d15-3da05e0fa17b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.545003 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.544881 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea08144-1c62-42ec-8d15-3da05e0fa17b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.545003 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.544909 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea08144-1c62-42ec-8d15-3da05e0fa17b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.545003 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.544935 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea08144-1c62-42ec-8d15-3da05e0fa17b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.545003 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.544957 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea08144-1c62-42ec-8d15-3da05e0fa17b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.545003 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.544973 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phpp9\" (UniqueName: \"kubernetes.io/projected/6ea08144-1c62-42ec-8d15-3da05e0fa17b-kube-api-access-phpp9\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.545003 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.544992 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea08144-1c62-42ec-8d15-3da05e0fa17b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.545283 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.545015 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea08144-1c62-42ec-8d15-3da05e0fa17b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.545283 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.545049 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea08144-1c62-42ec-8d15-3da05e0fa17b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.545542 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.545516 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea08144-1c62-42ec-8d15-3da05e0fa17b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.545643 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.545545 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea08144-1c62-42ec-8d15-3da05e0fa17b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.545643 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.545616 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea08144-1c62-42ec-8d15-3da05e0fa17b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.545722 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.545692 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea08144-1c62-42ec-8d15-3da05e0fa17b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.545807 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.545788 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea08144-1c62-42ec-8d15-3da05e0fa17b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.548035 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.548011 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea08144-1c62-42ec-8d15-3da05e0fa17b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.548191 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.548173 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea08144-1c62-42ec-8d15-3da05e0fa17b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.559079 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.559054 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phpp9\" (UniqueName: \"kubernetes.io/projected/6ea08144-1c62-42ec-8d15-3da05e0fa17b-kube-api-access-phpp9\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.587941 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.587909 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea08144-1c62-42ec-8d15-3da05e0fa17b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4j46q6\" (UID: \"6ea08144-1c62-42ec-8d15-3da05e0fa17b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.694233 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.694158 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:07.911617 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:07.911594 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6"] Jul 06 19:06:07.913155 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:06:07.913127 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea08144_1c62_42ec_8d15_3da05e0fa17b.slice/crio-371af24fb0a1f13038b20d9a1c851838f145855b0bd1610180f8f367a8af4ec3 WatchSource:0}: Error finding container 371af24fb0a1f13038b20d9a1c851838f145855b0bd1610180f8f367a8af4ec3: Status 404 returned error can't find the container with id 371af24fb0a1f13038b20d9a1c851838f145855b0bd1610180f8f367a8af4ec3 Jul 06 19:06:08.103584 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:08.103554 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:08.114750 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:08.114714 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" event={"ID":"6ea08144-1c62-42ec-8d15-3da05e0fa17b","Type":"ContainerStarted","Data":"371af24fb0a1f13038b20d9a1c851838f145855b0bd1610180f8f367a8af4ec3"} Jul 06 19:06:10.350116 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:10.350076 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:06:10.350394 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:10.350162 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:06:10.350394 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:10.350196 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:06:11.130793 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:11.130745 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" event={"ID":"6ea08144-1c62-42ec-8d15-3da05e0fa17b","Type":"ContainerStarted","Data":"ec7b09139ba82389ca3cef82ddb2b94316bbb5f853c2f2f1c29665c7b0a3dae2"} Jul 06 19:06:11.382192 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:11.382098 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" podStartSLOduration=1.947703979 podStartE2EDuration="4.38208377s" totalImagesPullingTime="2.434379791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:11.373641237 +0000 UTC m=+432.642823579" watchObservedRunningTime="2026-07-06 19:06:11.38208377 +0000 UTC m=+432.651266111" Jul 06 19:06:11.694547 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:11.694469 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:11.699515 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:11.699489 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:12.134522 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:12.134495 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:12.135306 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:12.135293 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4j46q6" Jul 06 19:06:13.095510 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:13.095482 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-5psq2" Jul 06 19:06:18.392842 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.392804 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn"] Jul 06 19:06:18.395485 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.395466 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz"] Jul 06 19:06:18.395619 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.395597 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:06:18.397208 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.397193 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:06:18.409321 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.409305 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 19:06:18.413712 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.413695 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8s2tk\"" Jul 06 19:06:18.413817 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.413775 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 06 19:06:18.414966 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.414949 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2zghc\"" Jul 06 19:06:18.535298 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.535264 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc5cn\" (UniqueName: \"kubernetes.io/projected/ad5da93d-e1e5-4854-98fb-7d8f95143b13-kube-api-access-qc5cn\") pod \"kserve-controller-manager-d6f4b65b8-gwbxn\" (UID: \"ad5da93d-e1e5-4854-98fb-7d8f95143b13\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:06:18.535485 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.535318 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad5da93d-e1e5-4854-98fb-7d8f95143b13-cert\") pod \"kserve-controller-manager-d6f4b65b8-gwbxn\" (UID: \"ad5da93d-e1e5-4854-98fb-7d8f95143b13\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:06:18.535485 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.535350 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba75546a-9e3d-4363-a68d-b194380ff105-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-sq8dz\" (UID: \"ba75546a-9e3d-4363-a68d-b194380ff105\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:06:18.535485 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.535385 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zpmq\" (UniqueName: \"kubernetes.io/projected/ba75546a-9e3d-4363-a68d-b194380ff105-kube-api-access-9zpmq\") pod \"llmisvc-controller-manager-777c6bd7d5-sq8dz\" (UID: \"ba75546a-9e3d-4363-a68d-b194380ff105\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:06:18.636674 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.636642 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc5cn\" (UniqueName: \"kubernetes.io/projected/ad5da93d-e1e5-4854-98fb-7d8f95143b13-kube-api-access-qc5cn\") pod \"kserve-controller-manager-d6f4b65b8-gwbxn\" (UID: \"ad5da93d-e1e5-4854-98fb-7d8f95143b13\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:06:18.636825 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.636699 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad5da93d-e1e5-4854-98fb-7d8f95143b13-cert\") pod \"kserve-controller-manager-d6f4b65b8-gwbxn\" (UID: \"ad5da93d-e1e5-4854-98fb-7d8f95143b13\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:06:18.636825 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.636726 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba75546a-9e3d-4363-a68d-b194380ff105-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-sq8dz\" (UID: \"ba75546a-9e3d-4363-a68d-b194380ff105\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:06:18.636825 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.636748 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zpmq\" (UniqueName: \"kubernetes.io/projected/ba75546a-9e3d-4363-a68d-b194380ff105-kube-api-access-9zpmq\") pod \"llmisvc-controller-manager-777c6bd7d5-sq8dz\" (UID: \"ba75546a-9e3d-4363-a68d-b194380ff105\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:06:18.639326 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.639298 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba75546a-9e3d-4363-a68d-b194380ff105-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-sq8dz\" (UID: \"ba75546a-9e3d-4363-a68d-b194380ff105\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:06:18.639456 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.639385 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad5da93d-e1e5-4854-98fb-7d8f95143b13-cert\") pod \"kserve-controller-manager-d6f4b65b8-gwbxn\" (UID: \"ad5da93d-e1e5-4854-98fb-7d8f95143b13\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:06:18.649576 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.649525 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc5cn\" (UniqueName: \"kubernetes.io/projected/ad5da93d-e1e5-4854-98fb-7d8f95143b13-kube-api-access-qc5cn\") pod \"kserve-controller-manager-d6f4b65b8-gwbxn\" (UID: \"ad5da93d-e1e5-4854-98fb-7d8f95143b13\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:06:18.663281 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.663261 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zpmq\" (UniqueName: \"kubernetes.io/projected/ba75546a-9e3d-4363-a68d-b194380ff105-kube-api-access-9zpmq\") pod \"llmisvc-controller-manager-777c6bd7d5-sq8dz\" (UID: \"ba75546a-9e3d-4363-a68d-b194380ff105\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:06:18.706253 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.706231 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:06:18.711447 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:18.711401 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:06:19.527499 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:19.527465 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz"] Jul 06 19:06:19.594210 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:19.594181 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn"] Jul 06 19:06:19.704526 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:06:19.704496 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podba75546a_9e3d_4363_a68d_b194380ff105.slice/crio-e5ee84a5f3075d143a4f3a7387c715357605aa4a4f75a28b4921cd3410002c8e WatchSource:0}: Error finding container e5ee84a5f3075d143a4f3a7387c715357605aa4a4f75a28b4921cd3410002c8e: Status 404 returned error can't find the container with id e5ee84a5f3075d143a4f3a7387c715357605aa4a4f75a28b4921cd3410002c8e Jul 06 19:06:19.705335 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:19.705308 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz"] Jul 06 19:06:19.884749 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:19.884711 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn"] Jul 06 19:06:19.887387 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:06:19.887361 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5da93d_e1e5_4854_98fb_7d8f95143b13.slice/crio-3456db609de4b860669e160f28e9a718eaac385cf3c01c594919fa18efc988c4 WatchSource:0}: Error finding container 3456db609de4b860669e160f28e9a718eaac385cf3c01c594919fa18efc988c4: Status 404 returned error can't find the container with id 3456db609de4b860669e160f28e9a718eaac385cf3c01c594919fa18efc988c4 Jul 06 19:06:20.167333 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:20.167249 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" event={"ID":"ba75546a-9e3d-4363-a68d-b194380ff105","Type":"ContainerStarted","Data":"e5ee84a5f3075d143a4f3a7387c715357605aa4a4f75a28b4921cd3410002c8e"} Jul 06 19:06:20.168443 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:20.168406 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" event={"ID":"ad5da93d-e1e5-4854-98fb-7d8f95143b13","Type":"ContainerStarted","Data":"3456db609de4b860669e160f28e9a718eaac385cf3c01c594919fa18efc988c4"} Jul 06 19:06:22.400549 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.400410 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5qwft"] Jul 06 19:06:22.401015 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.400693 2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" podUID="5e0ba8c8-2ff4-491d-bc37-4e68e088f071" containerName="discovery" containerID="cri-o://aa8b5e1b5a3c46becae1b8441ca9ce6d0bd0ca9dac0e6fe3e739ff783abdcaab" gracePeriod=30 Jul 06 19:06:22.651321 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.651245 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:22.773914 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.773880 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-token\") pod \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " Jul 06 19:06:22.774103 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.773931 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhktc\" (UniqueName: \"kubernetes.io/projected/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-kube-api-access-lhktc\") pod \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " Jul 06 19:06:22.774103 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.773966 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-kubeconfig\") pod \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " Jul 06 19:06:22.774222 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.774109 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-csr-dns-cert\") pod \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " Jul 06 19:06:22.774222 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.774172 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-local-certs\") pod \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " Jul 06 19:06:22.774320 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.774234 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-cacerts\") pod \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " Jul 06 19:06:22.774320 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.774269 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-csr-ca-configmap\") pod \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\" (UID: \"5e0ba8c8-2ff4-491d-bc37-4e68e088f071\") " Jul 06 19:06:22.774971 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.774928 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "5e0ba8c8-2ff4-491d-bc37-4e68e088f071" (UID: "5e0ba8c8-2ff4-491d-bc37-4e68e088f071"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:06:22.776997 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.776969 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "5e0ba8c8-2ff4-491d-bc37-4e68e088f071" (UID: "5e0ba8c8-2ff4-491d-bc37-4e68e088f071"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:06:22.777125 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.777093 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "5e0ba8c8-2ff4-491d-bc37-4e68e088f071" (UID: "5e0ba8c8-2ff4-491d-bc37-4e68e088f071"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:06:22.777242 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.777126 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e0ba8c8-2ff4-491d-bc37-4e68e088f071" (UID: "5e0ba8c8-2ff4-491d-bc37-4e68e088f071"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:06:22.777242 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.777150 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-kube-api-access-lhktc" (OuterVolumeSpecName: "kube-api-access-lhktc") pod "5e0ba8c8-2ff4-491d-bc37-4e68e088f071" (UID: "5e0ba8c8-2ff4-491d-bc37-4e68e088f071"). InnerVolumeSpecName "kube-api-access-lhktc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:06:22.777355 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.777330 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-local-certs" (OuterVolumeSpecName: "local-certs") pod "5e0ba8c8-2ff4-491d-bc37-4e68e088f071" (UID: "5e0ba8c8-2ff4-491d-bc37-4e68e088f071"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:06:22.777635 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.777604 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-cacerts" (OuterVolumeSpecName: "cacerts") pod "5e0ba8c8-2ff4-491d-bc37-4e68e088f071" (UID: "5e0ba8c8-2ff4-491d-bc37-4e68e088f071"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:06:22.875635 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.875598 2553 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-cacerts\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:06:22.875635 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.875627 2553 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-csr-ca-configmap\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:06:22.875635 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.875638 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-token\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:06:22.875876 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.875654 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhktc\" (UniqueName: \"kubernetes.io/projected/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-kube-api-access-lhktc\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:06:22.875876 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.875663 2553 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-kubeconfig\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:06:22.875876 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.875671 2553 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-istio-csr-dns-cert\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:06:22.875876 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:22.875679 2553 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e0ba8c8-2ff4-491d-bc37-4e68e088f071-local-certs\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:06:23.183777 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:23.183745 2553 generic.go:356] "Generic (PLEG): container finished" podID="5e0ba8c8-2ff4-491d-bc37-4e68e088f071" containerID="aa8b5e1b5a3c46becae1b8441ca9ce6d0bd0ca9dac0e6fe3e739ff783abdcaab" exitCode=0 Jul 06 19:06:23.183976 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:23.183798 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" Jul 06 19:06:23.183976 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:23.183841 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" event={"ID":"5e0ba8c8-2ff4-491d-bc37-4e68e088f071","Type":"ContainerDied","Data":"aa8b5e1b5a3c46becae1b8441ca9ce6d0bd0ca9dac0e6fe3e739ff783abdcaab"} Jul 06 19:06:23.183976 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:23.183901 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5qwft" event={"ID":"5e0ba8c8-2ff4-491d-bc37-4e68e088f071","Type":"ContainerDied","Data":"c8b0a3db87ddd2fab00c07eadf825f73bded83f661bcca57cc0bca4796ec0138"} Jul 06 19:06:23.183976 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:23.183932 2553 scope.go:117] "RemoveContainer" containerID="aa8b5e1b5a3c46becae1b8441ca9ce6d0bd0ca9dac0e6fe3e739ff783abdcaab" Jul 06 19:06:23.652278 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:23.652109 2553 scope.go:117] "RemoveContainer" containerID="aa8b5e1b5a3c46becae1b8441ca9ce6d0bd0ca9dac0e6fe3e739ff783abdcaab" Jul 06 19:06:23.652573 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:06:23.652507 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa8b5e1b5a3c46becae1b8441ca9ce6d0bd0ca9dac0e6fe3e739ff783abdcaab\": container with ID starting with aa8b5e1b5a3c46becae1b8441ca9ce6d0bd0ca9dac0e6fe3e739ff783abdcaab not found: ID does not exist" containerID="aa8b5e1b5a3c46becae1b8441ca9ce6d0bd0ca9dac0e6fe3e739ff783abdcaab" Jul 06 19:06:23.652573 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:23.652535 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa8b5e1b5a3c46becae1b8441ca9ce6d0bd0ca9dac0e6fe3e739ff783abdcaab"} err="failed to get container status \"aa8b5e1b5a3c46becae1b8441ca9ce6d0bd0ca9dac0e6fe3e739ff783abdcaab\": rpc error: code = NotFound desc = could not find container \"aa8b5e1b5a3c46becae1b8441ca9ce6d0bd0ca9dac0e6fe3e739ff783abdcaab\": container with ID starting with aa8b5e1b5a3c46becae1b8441ca9ce6d0bd0ca9dac0e6fe3e739ff783abdcaab not found: ID does not exist" Jul 06 19:06:23.771407 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:23.771369 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5qwft"] Jul 06 19:06:23.786614 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:23.786557 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5qwft"] Jul 06 19:06:25.194457 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:25.194404 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" event={"ID":"ba75546a-9e3d-4363-a68d-b194380ff105","Type":"ContainerStarted","Data":"1b021fb3585448a2f827c409396d923b786991270bbebae9e776c807e826efe0"} Jul 06 19:06:25.194915 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:25.194528 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:06:25.196094 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:25.196072 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" event={"ID":"ad5da93d-e1e5-4854-98fb-7d8f95143b13","Type":"ContainerStarted","Data":"e36021cb33c2aafbfefd5d8b3b4fe44ebb838c8d94c95138a8464413bfa25619"} Jul 06 19:06:25.196202 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:25.196151 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:06:25.215284 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:25.215241 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" podStartSLOduration=3.689172239 podStartE2EDuration="8.2152277s" totalImagesPullingTime="4.526055461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:25.213905494 +0000 UTC m=+446.483087836" watchObservedRunningTime="2026-07-06 19:06:25.2152277 +0000 UTC m=+446.484410043" Jul 06 19:06:25.281723 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:25.281693 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e0ba8c8-2ff4-491d-bc37-4e68e088f071" path="/var/lib/kubelet/pods/5e0ba8c8-2ff4-491d-bc37-4e68e088f071/volumes" Jul 06 19:06:56.202191 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:56.202157 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:06:56.205227 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:56.205204 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:06:56.225013 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:06:56.224964 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" podStartSLOduration=34.879870902 podStartE2EDuration="39.224947562s" totalImagesPullingTime="4.34507666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:25.235396669 +0000 UTC m=+446.504579026" watchObservedRunningTime="2026-07-06 19:06:56.224947562 +0000 UTC m=+477.494129894" Jul 06 19:07:17.762928 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:17.762896 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vmxrv"] Jul 06 19:07:17.763352 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:17.763216 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e0ba8c8-2ff4-491d-bc37-4e68e088f071" containerName="discovery" Jul 06 19:07:17.763352 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:17.763227 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e0ba8c8-2ff4-491d-bc37-4e68e088f071" containerName="discovery" Jul 06 19:07:17.763352 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:17.763288 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e0ba8c8-2ff4-491d-bc37-4e68e088f071" containerName="discovery" Jul 06 19:07:17.768555 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:17.768534 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vmxrv" Jul 06 19:07:17.773033 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:07:17.773005 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kuadrant-operator-catalog-dockercfg-72f76\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-72f76\"" type="*v1.Secret" Jul 06 19:07:17.774678 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:07:17.774644 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-catalog-vmxrv\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="a4c8598b-a456-4ac4-a9aa-24de2db43e33" pod="kuadrant-system/kuadrant-operator-catalog-vmxrv" Jul 06 19:07:17.774800 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:07:17.774672 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 06 19:07:17.774800 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:07:17.774674 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 06 19:07:17.786408 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:17.786377 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vmxrv"] Jul 06 19:07:17.908677 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:17.908647 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh448\" (UniqueName: \"kubernetes.io/projected/a4c8598b-a456-4ac4-a9aa-24de2db43e33-kube-api-access-fh448\") pod \"kuadrant-operator-catalog-vmxrv\" (UID: \"a4c8598b-a456-4ac4-a9aa-24de2db43e33\") " pod="kuadrant-system/kuadrant-operator-catalog-vmxrv" Jul 06 19:07:18.010021 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:18.009976 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh448\" (UniqueName: \"kubernetes.io/projected/a4c8598b-a456-4ac4-a9aa-24de2db43e33-kube-api-access-fh448\") pod \"kuadrant-operator-catalog-vmxrv\" (UID: \"a4c8598b-a456-4ac4-a9aa-24de2db43e33\") " pod="kuadrant-system/kuadrant-operator-catalog-vmxrv" Jul 06 19:07:18.019926 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:18.019860 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vmxrv"] Jul 06 19:07:18.020118 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:07:18.020098 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fh448], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-catalog-vmxrv" podUID="a4c8598b-a456-4ac4-a9aa-24de2db43e33" Jul 06 19:07:18.237754 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:18.237715 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6tkch"] Jul 06 19:07:18.239802 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:18.239780 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6tkch" Jul 06 19:07:18.260238 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:18.260211 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6tkch"] Jul 06 19:07:18.313121 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:18.313013 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pjm9\" (UniqueName: \"kubernetes.io/projected/5c49682e-63e9-4178-91ca-a2021cc916ed-kube-api-access-7pjm9\") pod \"kuadrant-operator-catalog-6tkch\" (UID: \"5c49682e-63e9-4178-91ca-a2021cc916ed\") " pod="kuadrant-system/kuadrant-operator-catalog-6tkch" Jul 06 19:07:18.415387 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:18.415341 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pjm9\" (UniqueName: \"kubernetes.io/projected/5c49682e-63e9-4178-91ca-a2021cc916ed-kube-api-access-7pjm9\") pod \"kuadrant-operator-catalog-6tkch\" (UID: \"5c49682e-63e9-4178-91ca-a2021cc916ed\") " pod="kuadrant-system/kuadrant-operator-catalog-6tkch" Jul 06 19:07:18.417778 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:18.417749 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vmxrv" Jul 06 19:07:18.428272 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:18.428255 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vmxrv" Jul 06 19:07:18.733994 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:18.733966 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:07:18.773112 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:18.773083 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:07:18.779436 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:18.779388 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh448\" (UniqueName: \"kubernetes.io/projected/a4c8598b-a456-4ac4-a9aa-24de2db43e33-kube-api-access-fh448\") pod \"kuadrant-operator-catalog-vmxrv\" (UID: \"a4c8598b-a456-4ac4-a9aa-24de2db43e33\") " pod="kuadrant-system/kuadrant-operator-catalog-vmxrv" Jul 06 19:07:18.783630 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:18.783608 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pjm9\" (UniqueName: \"kubernetes.io/projected/5c49682e-63e9-4178-91ca-a2021cc916ed-kube-api-access-7pjm9\") pod \"kuadrant-operator-catalog-6tkch\" (UID: \"5c49682e-63e9-4178-91ca-a2021cc916ed\") " pod="kuadrant-system/kuadrant-operator-catalog-6tkch" Jul 06 19:07:18.918698 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:18.918668 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh448\" (UniqueName: \"kubernetes.io/projected/a4c8598b-a456-4ac4-a9aa-24de2db43e33-kube-api-access-fh448\") pod \"a4c8598b-a456-4ac4-a9aa-24de2db43e33\" (UID: \"a4c8598b-a456-4ac4-a9aa-24de2db43e33\") " Jul 06 19:07:18.920874 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:18.920841 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c8598b-a456-4ac4-a9aa-24de2db43e33-kube-api-access-fh448" (OuterVolumeSpecName: "kube-api-access-fh448") pod "a4c8598b-a456-4ac4-a9aa-24de2db43e33" (UID: "a4c8598b-a456-4ac4-a9aa-24de2db43e33"). InnerVolumeSpecName "kube-api-access-fh448". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:07:19.019722 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:19.019652 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh448\" (UniqueName: \"kubernetes.io/projected/a4c8598b-a456-4ac4-a9aa-24de2db43e33-kube-api-access-fh448\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:07:19.254251 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:19.254218 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-72f76\"" Jul 06 19:07:19.259103 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:19.259089 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6tkch" Jul 06 19:07:19.414181 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:19.414152 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6tkch"] Jul 06 19:07:19.417511 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:07:19.417481 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c49682e_63e9_4178_91ca_a2021cc916ed.slice/crio-94d49626c2192e4054b5528dafb8b89522b2fb1e1ed7f9d2f1c41301893d79a7 WatchSource:0}: Error finding container 94d49626c2192e4054b5528dafb8b89522b2fb1e1ed7f9d2f1c41301893d79a7: Status 404 returned error can't find the container with id 94d49626c2192e4054b5528dafb8b89522b2fb1e1ed7f9d2f1c41301893d79a7 Jul 06 19:07:19.425541 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:19.425524 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vmxrv" Jul 06 19:07:19.425541 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:19.425519 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6tkch" event={"ID":"5c49682e-63e9-4178-91ca-a2021cc916ed","Type":"ContainerStarted","Data":"94d49626c2192e4054b5528dafb8b89522b2fb1e1ed7f9d2f1c41301893d79a7"} Jul 06 19:07:19.521373 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:19.521341 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vmxrv"] Jul 06 19:07:19.543856 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:19.543799 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vmxrv"] Jul 06 19:07:21.282615 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:21.282580 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4c8598b-a456-4ac4-a9aa-24de2db43e33" path="/var/lib/kubelet/pods/a4c8598b-a456-4ac4-a9aa-24de2db43e33/volumes" Jul 06 19:07:22.439633 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:22.439599 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6tkch" event={"ID":"5c49682e-63e9-4178-91ca-a2021cc916ed","Type":"ContainerStarted","Data":"ff11478bc35e735a6a1ae4a068591bfac0cfeb307c6cae3aada265d565b2964d"} Jul 06 19:07:29.260154 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:29.260119 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-6tkch" Jul 06 19:07:29.260669 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:29.260177 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-6tkch" Jul 06 19:07:29.283305 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:29.283278 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-6tkch" Jul 06 19:07:29.315141 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:29.315094 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-6tkch" podStartSLOduration=9.026644458 podStartE2EDuration="11.315078967s" totalImagesPullingTime="2.288434509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:07:22.464978421 +0000 UTC m=+503.734160763" watchObservedRunningTime="2026-07-06 19:07:29.315078967 +0000 UTC m=+510.584261344" Jul 06 19:07:29.494816 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:29.494783 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-6tkch" Jul 06 19:07:49.842895 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:49.842856 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt"] Jul 06 19:07:49.846443 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:49.846392 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" Jul 06 19:07:49.849119 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:49.849097 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 06 19:07:49.849201 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:49.849097 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-x68mr\"" Jul 06 19:07:49.857775 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:49.857752 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlgpx\" (UniqueName: \"kubernetes.io/projected/632d5de0-2e1b-4124-906e-21a096713b69-kube-api-access-xlgpx\") pod \"dns-operator-controller-manager-864dd78c4c-wklwt\" (UID: \"632d5de0-2e1b-4124-906e-21a096713b69\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" Jul 06 19:07:49.931988 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:49.931949 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt"] Jul 06 19:07:49.958340 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:49.958310 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlgpx\" (UniqueName: \"kubernetes.io/projected/632d5de0-2e1b-4124-906e-21a096713b69-kube-api-access-xlgpx\") pod \"dns-operator-controller-manager-864dd78c4c-wklwt\" (UID: \"632d5de0-2e1b-4124-906e-21a096713b69\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" Jul 06 19:07:49.970130 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:49.970099 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlgpx\" (UniqueName: \"kubernetes.io/projected/632d5de0-2e1b-4124-906e-21a096713b69-kube-api-access-xlgpx\") pod \"dns-operator-controller-manager-864dd78c4c-wklwt\" (UID: \"632d5de0-2e1b-4124-906e-21a096713b69\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" Jul 06 19:07:50.156483 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:50.156388 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" Jul 06 19:07:50.306462 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:50.306437 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt"] Jul 06 19:07:50.309095 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:07:50.309071 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod632d5de0_2e1b_4124_906e_21a096713b69.slice/crio-df417b9243a566319b91e2c13a137bf39c0603a331c70d0ebc662c7c021cf634 WatchSource:0}: Error finding container df417b9243a566319b91e2c13a137bf39c0603a331c70d0ebc662c7c021cf634: Status 404 returned error can't find the container with id df417b9243a566319b91e2c13a137bf39c0603a331c70d0ebc662c7c021cf634 Jul 06 19:07:50.560299 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:50.560269 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" event={"ID":"632d5de0-2e1b-4124-906e-21a096713b69","Type":"ContainerStarted","Data":"df417b9243a566319b91e2c13a137bf39c0603a331c70d0ebc662c7c021cf634"} Jul 06 19:07:53.574686 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:53.574637 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" event={"ID":"632d5de0-2e1b-4124-906e-21a096713b69","Type":"ContainerStarted","Data":"bb81e665dc41db2d20aa1ae59107d9dcc1eaf39606e5bedec62700c9a2a9de12"} Jul 06 19:07:53.575056 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:53.574747 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" Jul 06 19:07:53.630323 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:53.630282 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" podStartSLOduration=2.274326325 podStartE2EDuration="4.630267867s" totalImagesPullingTime="2.355941542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:07:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:07:53.628406472 +0000 UTC m=+534.897588825" watchObservedRunningTime="2026-07-06 19:07:53.630267867 +0000 UTC m=+534.899450209" Jul 06 19:07:55.048333 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:55.048297 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55"] Jul 06 19:07:55.051681 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:55.051666 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:07:55.053919 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:07:55.053891 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k6m55\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="8ee1d682-13e1-450b-a608-799a7201a97d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:07:55.059133 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:07:55.059112 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kuadrant-operator-controller-manager-dockercfg-xmrqf\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xmrqf\"" type="*v1.Secret" Jul 06 19:07:55.093990 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:55.093970 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ee1d682-13e1-450b-a608-799a7201a97d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k6m55\" (UID: \"8ee1d682-13e1-450b-a608-799a7201a97d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:07:55.094084 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:55.094004 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfqrh\" (UniqueName: \"kubernetes.io/projected/8ee1d682-13e1-450b-a608-799a7201a97d-kube-api-access-cfqrh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k6m55\" (UID: \"8ee1d682-13e1-450b-a608-799a7201a97d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:07:55.118214 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:55.118189 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55"] Jul 06 19:07:55.194787 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:55.194765 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ee1d682-13e1-450b-a608-799a7201a97d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k6m55\" (UID: \"8ee1d682-13e1-450b-a608-799a7201a97d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:07:55.194897 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:55.194801 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfqrh\" (UniqueName: \"kubernetes.io/projected/8ee1d682-13e1-450b-a608-799a7201a97d-kube-api-access-cfqrh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k6m55\" (UID: \"8ee1d682-13e1-450b-a608-799a7201a97d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:07:55.195170 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:55.195152 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ee1d682-13e1-450b-a608-799a7201a97d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k6m55\" (UID: \"8ee1d682-13e1-450b-a608-799a7201a97d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:07:55.210675 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:55.210658 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfqrh\" (UniqueName: \"kubernetes.io/projected/8ee1d682-13e1-450b-a608-799a7201a97d-kube-api-access-cfqrh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k6m55\" (UID: \"8ee1d682-13e1-450b-a608-799a7201a97d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:07:56.361376 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:56.361342 2553 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" secret="" err="failed to sync secret cache: timed out waiting for the condition" Jul 06 19:07:56.361769 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:56.361451 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:07:56.520569 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:56.520531 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55"] Jul 06 19:07:56.525027 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:07:56.524997 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee1d682_13e1_450b_a608_799a7201a97d.slice/crio-6cedc694b5a38e872549827e8cc9819adfd139b4fc25b9abb1292dd41f0a8a0c WatchSource:0}: Error finding container 6cedc694b5a38e872549827e8cc9819adfd139b4fc25b9abb1292dd41f0a8a0c: Status 404 returned error can't find the container with id 6cedc694b5a38e872549827e8cc9819adfd139b4fc25b9abb1292dd41f0a8a0c Jul 06 19:07:56.551926 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:56.551903 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xmrqf\"" Jul 06 19:07:56.588991 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:56.588960 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" event={"ID":"8ee1d682-13e1-450b-a608-799a7201a97d","Type":"ContainerStarted","Data":"6cedc694b5a38e872549827e8cc9819adfd139b4fc25b9abb1292dd41f0a8a0c"} Jul 06 19:07:58.466168 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:58.466131 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr"] Jul 06 19:07:58.469876 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:58.469854 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" Jul 06 19:07:58.472383 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:58.472365 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-mvgvw\"" Jul 06 19:07:58.486696 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:58.486671 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr"] Jul 06 19:07:58.525620 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:58.525590 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8sr6\" (UniqueName: \"kubernetes.io/projected/a595072e-bd51-4664-b6b5-1cc6b1c778de-kube-api-access-v8sr6\") pod \"limitador-operator-controller-manager-859d7b78bd-lqpnr\" (UID: \"a595072e-bd51-4664-b6b5-1cc6b1c778de\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" Jul 06 19:07:58.626557 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:58.626522 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8sr6\" (UniqueName: \"kubernetes.io/projected/a595072e-bd51-4664-b6b5-1cc6b1c778de-kube-api-access-v8sr6\") pod \"limitador-operator-controller-manager-859d7b78bd-lqpnr\" (UID: \"a595072e-bd51-4664-b6b5-1cc6b1c778de\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" Jul 06 19:07:58.636901 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:58.636878 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8sr6\" (UniqueName: \"kubernetes.io/projected/a595072e-bd51-4664-b6b5-1cc6b1c778de-kube-api-access-v8sr6\") pod \"limitador-operator-controller-manager-859d7b78bd-lqpnr\" (UID: \"a595072e-bd51-4664-b6b5-1cc6b1c778de\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" Jul 06 19:07:58.779540 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:58.779477 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" Jul 06 19:07:58.937725 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:58.937688 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr"] Jul 06 19:07:58.941766 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:07:58.941658 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda595072e_bd51_4664_b6b5_1cc6b1c778de.slice/crio-91b30b0b3895ba9fd6906866898d610668506376d1cc947ae82af798f87ca29e WatchSource:0}: Error finding container 91b30b0b3895ba9fd6906866898d610668506376d1cc947ae82af798f87ca29e: Status 404 returned error can't find the container with id 91b30b0b3895ba9fd6906866898d610668506376d1cc947ae82af798f87ca29e Jul 06 19:07:59.604781 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:07:59.604749 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" event={"ID":"a595072e-bd51-4664-b6b5-1cc6b1c778de","Type":"ContainerStarted","Data":"91b30b0b3895ba9fd6906866898d610668506376d1cc947ae82af798f87ca29e"} Jul 06 19:08:01.470877 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:01.470848 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9"] Jul 06 19:08:01.474094 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:01.474074 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" Jul 06 19:08:01.476117 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:01.476092 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-qp448\"" Jul 06 19:08:01.489128 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:01.489107 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9"] Jul 06 19:08:01.548763 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:01.548739 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6jgt\" (UniqueName: \"kubernetes.io/projected/2ab81f98-e81a-4b24-beae-8babf214f1fd-kube-api-access-d6jgt\") pod \"authorino-operator-75cc8d5cb7-kr5j9\" (UID: \"2ab81f98-e81a-4b24-beae-8babf214f1fd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" Jul 06 19:08:01.620461 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:01.615949 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" event={"ID":"a595072e-bd51-4664-b6b5-1cc6b1c778de","Type":"ContainerStarted","Data":"43b6b62a6a5ee374c9f0161528e7100aeedda197e90679d3f7b2e36e3db7d602"} Jul 06 19:08:01.620461 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:01.616894 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" Jul 06 19:08:01.646956 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:01.646909 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" podStartSLOduration=1.736446483 podStartE2EDuration="3.646894975s" totalImagesPullingTime="1.910448492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:08:01.646459408 +0000 UTC m=+542.915641752" watchObservedRunningTime="2026-07-06 19:08:01.646894975 +0000 UTC m=+542.916077382" Jul 06 19:08:01.649765 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:01.649739 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6jgt\" (UniqueName: \"kubernetes.io/projected/2ab81f98-e81a-4b24-beae-8babf214f1fd-kube-api-access-d6jgt\") pod \"authorino-operator-75cc8d5cb7-kr5j9\" (UID: \"2ab81f98-e81a-4b24-beae-8babf214f1fd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" Jul 06 19:08:01.661563 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:01.661540 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6jgt\" (UniqueName: \"kubernetes.io/projected/2ab81f98-e81a-4b24-beae-8babf214f1fd-kube-api-access-d6jgt\") pod \"authorino-operator-75cc8d5cb7-kr5j9\" (UID: \"2ab81f98-e81a-4b24-beae-8babf214f1fd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" Jul 06 19:08:01.783474 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:01.783379 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" Jul 06 19:08:02.940367 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:02.940272 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9"] Jul 06 19:08:02.943040 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:08:02.943014 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab81f98_e81a_4b24_beae_8babf214f1fd.slice/crio-89b672eadf86167bb3c60718bb6f3374d4ed1a3b53d243c58e0f882433600922 WatchSource:0}: Error finding container 89b672eadf86167bb3c60718bb6f3374d4ed1a3b53d243c58e0f882433600922: Status 404 returned error can't find the container with id 89b672eadf86167bb3c60718bb6f3374d4ed1a3b53d243c58e0f882433600922 Jul 06 19:08:03.627970 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:03.627934 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" event={"ID":"8ee1d682-13e1-450b-a608-799a7201a97d","Type":"ContainerStarted","Data":"b9ffad81e75ccef02425defb81aa741a4cc6ec62960f43025e044f16fb384d2a"} Jul 06 19:08:03.628168 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:03.628104 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:08:03.629168 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:03.629144 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" event={"ID":"2ab81f98-e81a-4b24-beae-8babf214f1fd","Type":"ContainerStarted","Data":"89b672eadf86167bb3c60718bb6f3374d4ed1a3b53d243c58e0f882433600922"} Jul 06 19:08:03.665330 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:03.665281 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" podStartSLOduration=2.337111579 podStartE2EDuration="8.665267058s" totalImagesPullingTime="6.328155479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:07:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:08:03.661938824 +0000 UTC m=+544.931121354" watchObservedRunningTime="2026-07-06 19:08:03.665267058 +0000 UTC m=+544.934449466" Jul 06 19:08:04.581312 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:04.581284 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" Jul 06 19:08:06.244633 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.244604 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vcvtg"] Jul 06 19:08:06.247779 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.247762 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vcvtg" Jul 06 19:08:06.253190 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.253171 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 06 19:08:06.253457 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.253441 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 06 19:08:06.255524 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.255506 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9xf9k\"" Jul 06 19:08:06.273442 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.273403 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vcvtg"] Jul 06 19:08:06.289301 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.289280 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec841e47-d3ca-43bf-a062-361f912c5f27-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vcvtg\" (UID: \"ec841e47-d3ca-43bf-a062-361f912c5f27\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vcvtg" Jul 06 19:08:06.289467 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.289316 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec841e47-d3ca-43bf-a062-361f912c5f27-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vcvtg\" (UID: \"ec841e47-d3ca-43bf-a062-361f912c5f27\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vcvtg" Jul 06 19:08:06.289467 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.289342 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spgbd\" (UniqueName: \"kubernetes.io/projected/ec841e47-d3ca-43bf-a062-361f912c5f27-kube-api-access-spgbd\") pod \"kuadrant-console-plugin-66b5774985-vcvtg\" (UID: \"ec841e47-d3ca-43bf-a062-361f912c5f27\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vcvtg" Jul 06 19:08:06.390395 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.390316 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec841e47-d3ca-43bf-a062-361f912c5f27-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vcvtg\" (UID: \"ec841e47-d3ca-43bf-a062-361f912c5f27\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vcvtg" Jul 06 19:08:06.390395 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.390357 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec841e47-d3ca-43bf-a062-361f912c5f27-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vcvtg\" (UID: \"ec841e47-d3ca-43bf-a062-361f912c5f27\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vcvtg" Jul 06 19:08:06.390395 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.390384 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spgbd\" (UniqueName: \"kubernetes.io/projected/ec841e47-d3ca-43bf-a062-361f912c5f27-kube-api-access-spgbd\") pod \"kuadrant-console-plugin-66b5774985-vcvtg\" (UID: \"ec841e47-d3ca-43bf-a062-361f912c5f27\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vcvtg" Jul 06 19:08:06.391058 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.391040 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec841e47-d3ca-43bf-a062-361f912c5f27-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-vcvtg\" (UID: \"ec841e47-d3ca-43bf-a062-361f912c5f27\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vcvtg" Jul 06 19:08:06.392960 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.392943 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec841e47-d3ca-43bf-a062-361f912c5f27-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-vcvtg\" (UID: \"ec841e47-d3ca-43bf-a062-361f912c5f27\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vcvtg" Jul 06 19:08:06.422068 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.422044 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spgbd\" (UniqueName: \"kubernetes.io/projected/ec841e47-d3ca-43bf-a062-361f912c5f27-kube-api-access-spgbd\") pod \"kuadrant-console-plugin-66b5774985-vcvtg\" (UID: \"ec841e47-d3ca-43bf-a062-361f912c5f27\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vcvtg" Jul 06 19:08:06.556380 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.556341 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vcvtg" Jul 06 19:08:06.647303 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.647265 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" event={"ID":"2ab81f98-e81a-4b24-beae-8babf214f1fd","Type":"ContainerStarted","Data":"deb554c8ea8e9d64e25a5590a2e44cf8ffa1ff75566ac7956d0185600c8d1264"} Jul 06 19:08:06.647488 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.647400 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" Jul 06 19:08:06.675992 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.675943 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" podStartSLOduration=2.5420997119999997 podStartE2EDuration="5.67592845s" totalImagesPullingTime="3.133828738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:08:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:08:06.674446546 +0000 UTC m=+547.943628884" watchObservedRunningTime="2026-07-06 19:08:06.67592845 +0000 UTC m=+547.945110792" Jul 06 19:08:06.728004 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:06.727981 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-vcvtg"] Jul 06 19:08:06.730569 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:08:06.730544 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec841e47_d3ca_43bf_a062_361f912c5f27.slice/crio-b25f5deb7a35dd9e33076f319d6d47c660dbe2394620048ac712c4e75b8aebcf WatchSource:0}: Error finding container b25f5deb7a35dd9e33076f319d6d47c660dbe2394620048ac712c4e75b8aebcf: Status 404 returned error can't find the container with id b25f5deb7a35dd9e33076f319d6d47c660dbe2394620048ac712c4e75b8aebcf Jul 06 19:08:07.654552 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:07.654517 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vcvtg" event={"ID":"ec841e47-d3ca-43bf-a062-361f912c5f27","Type":"ContainerStarted","Data":"b25f5deb7a35dd9e33076f319d6d47c660dbe2394620048ac712c4e75b8aebcf"} Jul 06 19:08:13.631109 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:13.631081 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" Jul 06 19:08:14.638273 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:14.638243 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:08:17.342209 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.342180 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55"] Jul 06 19:08:17.342659 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.342346 2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" podUID="8ee1d682-13e1-450b-a608-799a7201a97d" containerName="manager" containerID="cri-o://b9ffad81e75ccef02425defb81aa741a4cc6ec62960f43025e044f16fb384d2a" gracePeriod=2 Jul 06 19:08:17.397861 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.397829 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55"] Jul 06 19:08:17.410886 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.410858 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr"] Jul 06 19:08:17.411107 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.411061 2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" podUID="a595072e-bd51-4664-b6b5-1cc6b1c778de" containerName="manager" containerID="cri-o://43b6b62a6a5ee374c9f0161528e7100aeedda197e90679d3f7b2e36e3db7d602" gracePeriod=2 Jul 06 19:08:17.412593 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.412566 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8"] Jul 06 19:08:17.412957 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.412941 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ee1d682-13e1-450b-a608-799a7201a97d" containerName="manager" Jul 06 19:08:17.412957 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.412957 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ee1d682-13e1-450b-a608-799a7201a97d" containerName="manager" Jul 06 19:08:17.413085 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.413024 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ee1d682-13e1-450b-a608-799a7201a97d" containerName="manager" Jul 06 19:08:17.416046 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.416008 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8" Jul 06 19:08:17.433503 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:08:17.433476 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k6m55\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="8ee1d682-13e1-450b-a608-799a7201a97d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:08:17.444015 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.443986 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr"] Jul 06 19:08:17.463958 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:08:17.463923 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lqpnr\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="a595072e-bd51-4664-b6b5-1cc6b1c778de" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" Jul 06 19:08:17.476077 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:08:17.476048 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lqpnr\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="a595072e-bd51-4664-b6b5-1cc6b1c778de" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" Jul 06 19:08:17.486937 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.486912 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa414-a92b-4531-9ec9-5603bd025d65-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qhfr8\" (UID: \"3b1aa414-a92b-4531-9ec9-5603bd025d65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8" Jul 06 19:08:17.487019 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.486946 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7mrj\" (UniqueName: \"kubernetes.io/projected/3b1aa414-a92b-4531-9ec9-5603bd025d65-kube-api-access-c7mrj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qhfr8\" (UID: \"3b1aa414-a92b-4531-9ec9-5603bd025d65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8" Jul 06 19:08:17.526339 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.526183 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8"] Jul 06 19:08:17.588534 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.588499 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa414-a92b-4531-9ec9-5603bd025d65-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qhfr8\" (UID: \"3b1aa414-a92b-4531-9ec9-5603bd025d65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8" Jul 06 19:08:17.588677 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.588549 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7mrj\" (UniqueName: \"kubernetes.io/projected/3b1aa414-a92b-4531-9ec9-5603bd025d65-kube-api-access-c7mrj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qhfr8\" (UID: \"3b1aa414-a92b-4531-9ec9-5603bd025d65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8" Jul 06 19:08:17.588982 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.588944 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa414-a92b-4531-9ec9-5603bd025d65-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qhfr8\" (UID: \"3b1aa414-a92b-4531-9ec9-5603bd025d65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8" Jul 06 19:08:17.613502 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:08:17.613457 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k6m55\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="8ee1d682-13e1-450b-a608-799a7201a97d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:08:17.616243 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.615964 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7mrj\" (UniqueName: \"kubernetes.io/projected/3b1aa414-a92b-4531-9ec9-5603bd025d65-kube-api-access-c7mrj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qhfr8\" (UID: \"3b1aa414-a92b-4531-9ec9-5603bd025d65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8" Jul 06 19:08:17.657194 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.657169 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" Jul 06 19:08:17.659564 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:08:17.659532 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lqpnr\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="a595072e-bd51-4664-b6b5-1cc6b1c778de" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" Jul 06 19:08:17.661649 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:08:17.661624 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k6m55\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="8ee1d682-13e1-450b-a608-799a7201a97d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:08:17.673484 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.673468 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:08:17.690029 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.690009 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" Jul 06 19:08:17.704216 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.704190 2553 generic.go:356] "Generic (PLEG): container finished" podID="8ee1d682-13e1-450b-a608-799a7201a97d" containerID="b9ffad81e75ccef02425defb81aa741a4cc6ec62960f43025e044f16fb384d2a" exitCode=0 Jul 06 19:08:17.704302 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.704230 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:08:17.704302 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.704275 2553 scope.go:117] "RemoveContainer" containerID="b9ffad81e75ccef02425defb81aa741a4cc6ec62960f43025e044f16fb384d2a" Jul 06 19:08:17.705768 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.705750 2553 generic.go:356] "Generic (PLEG): container finished" podID="a595072e-bd51-4664-b6b5-1cc6b1c778de" containerID="43b6b62a6a5ee374c9f0161528e7100aeedda197e90679d3f7b2e36e3db7d602" exitCode=0 Jul 06 19:08:17.705868 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.705794 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" Jul 06 19:08:17.712916 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.712899 2553 scope.go:117] "RemoveContainer" containerID="b9ffad81e75ccef02425defb81aa741a4cc6ec62960f43025e044f16fb384d2a" Jul 06 19:08:17.713185 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:08:17.713168 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9ffad81e75ccef02425defb81aa741a4cc6ec62960f43025e044f16fb384d2a\": container with ID starting with b9ffad81e75ccef02425defb81aa741a4cc6ec62960f43025e044f16fb384d2a not found: ID does not exist" containerID="b9ffad81e75ccef02425defb81aa741a4cc6ec62960f43025e044f16fb384d2a" Jul 06 19:08:17.713233 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.713195 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9ffad81e75ccef02425defb81aa741a4cc6ec62960f43025e044f16fb384d2a"} err="failed to get container status \"b9ffad81e75ccef02425defb81aa741a4cc6ec62960f43025e044f16fb384d2a\": rpc error: code = NotFound desc = could not find container \"b9ffad81e75ccef02425defb81aa741a4cc6ec62960f43025e044f16fb384d2a\": container with ID starting with b9ffad81e75ccef02425defb81aa741a4cc6ec62960f43025e044f16fb384d2a not found: ID does not exist" Jul 06 19:08:17.713233 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.713213 2553 scope.go:117] "RemoveContainer" containerID="43b6b62a6a5ee374c9f0161528e7100aeedda197e90679d3f7b2e36e3db7d602" Jul 06 19:08:17.720894 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.720876 2553 scope.go:117] "RemoveContainer" containerID="43b6b62a6a5ee374c9f0161528e7100aeedda197e90679d3f7b2e36e3db7d602" Jul 06 19:08:17.721143 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:08:17.721126 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b6b62a6a5ee374c9f0161528e7100aeedda197e90679d3f7b2e36e3db7d602\": container with ID starting with 43b6b62a6a5ee374c9f0161528e7100aeedda197e90679d3f7b2e36e3db7d602 not found: ID does not exist" containerID="43b6b62a6a5ee374c9f0161528e7100aeedda197e90679d3f7b2e36e3db7d602" Jul 06 19:08:17.721185 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.721152 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b6b62a6a5ee374c9f0161528e7100aeedda197e90679d3f7b2e36e3db7d602"} err="failed to get container status \"43b6b62a6a5ee374c9f0161528e7100aeedda197e90679d3f7b2e36e3db7d602\": rpc error: code = NotFound desc = could not find container \"43b6b62a6a5ee374c9f0161528e7100aeedda197e90679d3f7b2e36e3db7d602\": container with ID starting with 43b6b62a6a5ee374c9f0161528e7100aeedda197e90679d3f7b2e36e3db7d602 not found: ID does not exist" Jul 06 19:08:17.789693 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.789674 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ee1d682-13e1-450b-a608-799a7201a97d-extensions-socket-volume\") pod \"8ee1d682-13e1-450b-a608-799a7201a97d\" (UID: \"8ee1d682-13e1-450b-a608-799a7201a97d\") " Jul 06 19:08:17.789816 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.789729 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfqrh\" (UniqueName: \"kubernetes.io/projected/8ee1d682-13e1-450b-a608-799a7201a97d-kube-api-access-cfqrh\") pod \"8ee1d682-13e1-450b-a608-799a7201a97d\" (UID: \"8ee1d682-13e1-450b-a608-799a7201a97d\") " Jul 06 19:08:17.789816 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.789764 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8sr6\" (UniqueName: \"kubernetes.io/projected/a595072e-bd51-4664-b6b5-1cc6b1c778de-kube-api-access-v8sr6\") pod \"a595072e-bd51-4664-b6b5-1cc6b1c778de\" (UID: \"a595072e-bd51-4664-b6b5-1cc6b1c778de\") " Jul 06 19:08:17.790207 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.790184 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee1d682-13e1-450b-a608-799a7201a97d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8ee1d682-13e1-450b-a608-799a7201a97d" (UID: "8ee1d682-13e1-450b-a608-799a7201a97d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:08:17.790297 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.790284 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8" Jul 06 19:08:17.791969 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.791945 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee1d682-13e1-450b-a608-799a7201a97d-kube-api-access-cfqrh" (OuterVolumeSpecName: "kube-api-access-cfqrh") pod "8ee1d682-13e1-450b-a608-799a7201a97d" (UID: "8ee1d682-13e1-450b-a608-799a7201a97d"). InnerVolumeSpecName "kube-api-access-cfqrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:08:17.792048 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.791974 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a595072e-bd51-4664-b6b5-1cc6b1c778de-kube-api-access-v8sr6" (OuterVolumeSpecName: "kube-api-access-v8sr6") pod "a595072e-bd51-4664-b6b5-1cc6b1c778de" (UID: "a595072e-bd51-4664-b6b5-1cc6b1c778de"). InnerVolumeSpecName "kube-api-access-v8sr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:08:17.815706 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:08:17.815673 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k6m55\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="8ee1d682-13e1-450b-a608-799a7201a97d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:08:17.818837 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:08:17.818806 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lqpnr\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="a595072e-bd51-4664-b6b5-1cc6b1c778de" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" Jul 06 19:08:17.891022 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.890994 2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ee1d682-13e1-450b-a608-799a7201a97d-extensions-socket-volume\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:08:17.891022 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.891023 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfqrh\" (UniqueName: \"kubernetes.io/projected/8ee1d682-13e1-450b-a608-799a7201a97d-kube-api-access-cfqrh\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:08:17.891213 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.891033 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8sr6\" (UniqueName: \"kubernetes.io/projected/a595072e-bd51-4664-b6b5-1cc6b1c778de-kube-api-access-v8sr6\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:08:17.989279 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:17.989191 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8"] Jul 06 19:08:17.992086 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:08:17.992061 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1aa414_a92b_4531_9ec9_5603bd025d65.slice/crio-718a469582641f79e295e599fde94939b03cc7da1a3c761eb80dcb1ef20d8346 WatchSource:0}: Error finding container 718a469582641f79e295e599fde94939b03cc7da1a3c761eb80dcb1ef20d8346: Status 404 returned error can't find the container with id 718a469582641f79e295e599fde94939b03cc7da1a3c761eb80dcb1ef20d8346 Jul 06 19:08:18.025180 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:08:18.025146 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lqpnr\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="a595072e-bd51-4664-b6b5-1cc6b1c778de" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" Jul 06 19:08:18.029640 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:08:18.029609 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k6m55\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="8ee1d682-13e1-450b-a608-799a7201a97d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:08:18.032441 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:08:18.032394 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lqpnr\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="a595072e-bd51-4664-b6b5-1cc6b1c778de" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" Jul 06 19:08:18.034590 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:08:18.034560 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k6m55\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="8ee1d682-13e1-450b-a608-799a7201a97d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:08:18.716487 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:18.716380 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8" event={"ID":"3b1aa414-a92b-4531-9ec9-5603bd025d65","Type":"ContainerStarted","Data":"3f4b94c732c1d45758f7500608666629ffa2fa5f6a141f594d7c27b56024a796"} Jul 06 19:08:18.716487 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:18.716462 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8" event={"ID":"3b1aa414-a92b-4531-9ec9-5603bd025d65","Type":"ContainerStarted","Data":"718a469582641f79e295e599fde94939b03cc7da1a3c761eb80dcb1ef20d8346"} Jul 06 19:08:18.716947 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:18.716494 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8" Jul 06 19:08:18.729656 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:08:18.729628 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lqpnr\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="a595072e-bd51-4664-b6b5-1cc6b1c778de" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" Jul 06 19:08:18.772833 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:18.772786 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8" podStartSLOduration=1.7727709040000001 podStartE2EDuration="1.772770904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:08:18.769934689 +0000 UTC m=+560.039117031" watchObservedRunningTime="2026-07-06 19:08:18.772770904 +0000 UTC m=+560.041953246" Jul 06 19:08:18.773308 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:08:18.773285 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k6m55\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="8ee1d682-13e1-450b-a608-799a7201a97d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:08:19.281962 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:08:19.281923 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lqpnr\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="a595072e-bd51-4664-b6b5-1cc6b1c778de" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lqpnr" Jul 06 19:08:19.284031 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:08:19.283989 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k6m55\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="8ee1d682-13e1-450b-a608-799a7201a97d" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k6m55" Jul 06 19:08:19.284512 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:19.284485 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ee1d682-13e1-450b-a608-799a7201a97d" path="/var/lib/kubelet/pods/8ee1d682-13e1-450b-a608-799a7201a97d/volumes" Jul 06 19:08:19.284939 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:19.284923 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a595072e-bd51-4664-b6b5-1cc6b1c778de" path="/var/lib/kubelet/pods/a595072e-bd51-4664-b6b5-1cc6b1c778de/volumes" Jul 06 19:08:22.260684 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:22.260643 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s"] Jul 06 19:08:22.261346 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:22.261074 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a595072e-bd51-4664-b6b5-1cc6b1c778de" containerName="manager" Jul 06 19:08:22.261346 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:22.261091 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a595072e-bd51-4664-b6b5-1cc6b1c778de" containerName="manager" Jul 06 19:08:22.261346 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:22.261176 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a595072e-bd51-4664-b6b5-1cc6b1c778de" containerName="manager" Jul 06 19:08:22.293049 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:22.293027 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:08:22.307458 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:22.307430 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s"] Jul 06 19:08:22.430580 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:22.430538 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f889fc14-3f1f-4bf4-ba6d-3f8e5f804413-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-khb2s\" (UID: \"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:08:22.430762 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:22.430605 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbx78\" (UniqueName: \"kubernetes.io/projected/f889fc14-3f1f-4bf4-ba6d-3f8e5f804413-kube-api-access-kbx78\") pod \"kuadrant-operator-controller-manager-86878c769f-khb2s\" (UID: \"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:08:22.531626 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:22.531535 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f889fc14-3f1f-4bf4-ba6d-3f8e5f804413-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-khb2s\" (UID: \"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:08:22.531803 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:22.531631 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbx78\" (UniqueName: \"kubernetes.io/projected/f889fc14-3f1f-4bf4-ba6d-3f8e5f804413-kube-api-access-kbx78\") pod \"kuadrant-operator-controller-manager-86878c769f-khb2s\" (UID: \"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:08:22.531990 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:22.531965 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f889fc14-3f1f-4bf4-ba6d-3f8e5f804413-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-khb2s\" (UID: \"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:08:22.545017 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:22.544991 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbx78\" (UniqueName: \"kubernetes.io/projected/f889fc14-3f1f-4bf4-ba6d-3f8e5f804413-kube-api-access-kbx78\") pod \"kuadrant-operator-controller-manager-86878c769f-khb2s\" (UID: \"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:08:22.605460 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:22.605378 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:08:29.724653 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:29.724626 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8" Jul 06 19:08:33.824379 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:33.824349 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s"] Jul 06 19:08:33.827634 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:08:33.827608 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf889fc14_3f1f_4bf4_ba6d_3f8e5f804413.slice/crio-2464173276dfd2980e24947f95d44570527535131f6ec96a37b292213c0ecd3b WatchSource:0}: Error finding container 2464173276dfd2980e24947f95d44570527535131f6ec96a37b292213c0ecd3b: Status 404 returned error can't find the container with id 2464173276dfd2980e24947f95d44570527535131f6ec96a37b292213c0ecd3b Jul 06 19:08:34.805761 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:34.805722 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vcvtg" event={"ID":"ec841e47-d3ca-43bf-a062-361f912c5f27","Type":"ContainerStarted","Data":"62cd337bca145e7825256b6842a17e4db59745e3a99858f5e1391db24e3d63cd"} Jul 06 19:08:34.807360 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:34.807335 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" event={"ID":"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413","Type":"ContainerStarted","Data":"8f17ee150552d55727ac712237cc31455f8a8d253c3a5f4a4c47d59c6f0d91a6"} Jul 06 19:08:34.807523 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:34.807408 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" event={"ID":"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413","Type":"ContainerStarted","Data":"2464173276dfd2980e24947f95d44570527535131f6ec96a37b292213c0ecd3b"} Jul 06 19:08:34.807523 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:34.807480 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:08:34.882969 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:34.882784 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-vcvtg" podStartSLOduration=1.3454727659999999 podStartE2EDuration="28.882761509s" totalImagesPullingTime="27.537288743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:08:34.839116705 +0000 UTC m=+576.108299047" watchObservedRunningTime="2026-07-06 19:08:34.882761509 +0000 UTC m=+576.151943853" Jul 06 19:08:34.884756 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:34.884711 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" podStartSLOduration=12.884692119 podStartE2EDuration="12.884692119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:08:34.881799797 +0000 UTC m=+576.150982137" watchObservedRunningTime="2026-07-06 19:08:34.884692119 +0000 UTC m=+576.153874463" Jul 06 19:08:45.815830 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:45.815798 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:08:45.928644 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:45.928615 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8"] Jul 06 19:08:45.928838 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:45.928817 2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8" podUID="3b1aa414-a92b-4531-9ec9-5603bd025d65" containerName="manager" containerID="cri-o://3f4b94c732c1d45758f7500608666629ffa2fa5f6a141f594d7c27b56024a796" gracePeriod=10 Jul 06 19:08:46.175593 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:46.175569 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8" Jul 06 19:08:46.237368 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:46.237334 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7mrj\" (UniqueName: \"kubernetes.io/projected/3b1aa414-a92b-4531-9ec9-5603bd025d65-kube-api-access-c7mrj\") pod \"3b1aa414-a92b-4531-9ec9-5603bd025d65\" (UID: \"3b1aa414-a92b-4531-9ec9-5603bd025d65\") " Jul 06 19:08:46.237565 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:46.237410 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa414-a92b-4531-9ec9-5603bd025d65-extensions-socket-volume\") pod \"3b1aa414-a92b-4531-9ec9-5603bd025d65\" (UID: \"3b1aa414-a92b-4531-9ec9-5603bd025d65\") " Jul 06 19:08:46.237790 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:46.237761 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1aa414-a92b-4531-9ec9-5603bd025d65-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3b1aa414-a92b-4531-9ec9-5603bd025d65" (UID: "3b1aa414-a92b-4531-9ec9-5603bd025d65"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:08:46.239825 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:46.239799 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1aa414-a92b-4531-9ec9-5603bd025d65-kube-api-access-c7mrj" (OuterVolumeSpecName: "kube-api-access-c7mrj") pod "3b1aa414-a92b-4531-9ec9-5603bd025d65" (UID: "3b1aa414-a92b-4531-9ec9-5603bd025d65"). InnerVolumeSpecName "kube-api-access-c7mrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:08:46.338010 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:46.337978 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7mrj\" (UniqueName: \"kubernetes.io/projected/3b1aa414-a92b-4531-9ec9-5603bd025d65-kube-api-access-c7mrj\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:08:46.338010 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:46.338005 2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b1aa414-a92b-4531-9ec9-5603bd025d65-extensions-socket-volume\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:08:46.864083 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:46.864048 2553 generic.go:356] "Generic (PLEG): container finished" podID="3b1aa414-a92b-4531-9ec9-5603bd025d65" containerID="3f4b94c732c1d45758f7500608666629ffa2fa5f6a141f594d7c27b56024a796" exitCode=0 Jul 06 19:08:46.864524 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:46.864093 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8" event={"ID":"3b1aa414-a92b-4531-9ec9-5603bd025d65","Type":"ContainerDied","Data":"3f4b94c732c1d45758f7500608666629ffa2fa5f6a141f594d7c27b56024a796"} Jul 06 19:08:46.864524 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:46.864107 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8" Jul 06 19:08:46.864524 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:46.864144 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8" event={"ID":"3b1aa414-a92b-4531-9ec9-5603bd025d65","Type":"ContainerDied","Data":"718a469582641f79e295e599fde94939b03cc7da1a3c761eb80dcb1ef20d8346"} Jul 06 19:08:46.864524 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:46.864169 2553 scope.go:117] "RemoveContainer" containerID="3f4b94c732c1d45758f7500608666629ffa2fa5f6a141f594d7c27b56024a796" Jul 06 19:08:46.872946 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:46.872930 2553 scope.go:117] "RemoveContainer" containerID="3f4b94c732c1d45758f7500608666629ffa2fa5f6a141f594d7c27b56024a796" Jul 06 19:08:46.873174 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:08:46.873157 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f4b94c732c1d45758f7500608666629ffa2fa5f6a141f594d7c27b56024a796\": container with ID starting with 3f4b94c732c1d45758f7500608666629ffa2fa5f6a141f594d7c27b56024a796 not found: ID does not exist" containerID="3f4b94c732c1d45758f7500608666629ffa2fa5f6a141f594d7c27b56024a796" Jul 06 19:08:46.873230 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:46.873183 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f4b94c732c1d45758f7500608666629ffa2fa5f6a141f594d7c27b56024a796"} err="failed to get container status \"3f4b94c732c1d45758f7500608666629ffa2fa5f6a141f594d7c27b56024a796\": rpc error: code = NotFound desc = could not find container \"3f4b94c732c1d45758f7500608666629ffa2fa5f6a141f594d7c27b56024a796\": container with ID starting with 3f4b94c732c1d45758f7500608666629ffa2fa5f6a141f594d7c27b56024a796 not found: ID does not exist" Jul 06 19:08:46.949919 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:46.949887 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8"] Jul 06 19:08:46.954458 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:46.954438 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qhfr8"] Jul 06 19:08:47.283273 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:47.283240 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b1aa414-a92b-4531-9ec9-5603bd025d65" path="/var/lib/kubelet/pods/3b1aa414-a92b-4531-9ec9-5603bd025d65/volumes" Jul 06 19:08:59.156526 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:59.156497 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:08:59.160899 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:59.160878 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:08:59.162944 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:59.162926 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:08:59.167640 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:08:59.167621 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:09:11.978348 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:11.978262 2553 generic.go:356] "Generic (PLEG): container finished" podID="ba75546a-9e3d-4363-a68d-b194380ff105" containerID="1b021fb3585448a2f827c409396d923b786991270bbebae9e776c807e826efe0" exitCode=1 Jul 06 19:09:11.978809 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:11.978337 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" event={"ID":"ba75546a-9e3d-4363-a68d-b194380ff105","Type":"ContainerDied","Data":"1b021fb3585448a2f827c409396d923b786991270bbebae9e776c807e826efe0"} Jul 06 19:09:11.978809 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:11.978720 2553 scope.go:117] "RemoveContainer" containerID="1b021fb3585448a2f827c409396d923b786991270bbebae9e776c807e826efe0" Jul 06 19:09:11.979978 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:11.979956 2553 generic.go:356] "Generic (PLEG): container finished" podID="ad5da93d-e1e5-4854-98fb-7d8f95143b13" containerID="e36021cb33c2aafbfefd5d8b3b4fe44ebb838c8d94c95138a8464413bfa25619" exitCode=1 Jul 06 19:09:11.980079 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:11.980028 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" event={"ID":"ad5da93d-e1e5-4854-98fb-7d8f95143b13","Type":"ContainerDied","Data":"e36021cb33c2aafbfefd5d8b3b4fe44ebb838c8d94c95138a8464413bfa25619"} Jul 06 19:09:11.980361 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:11.980341 2553 scope.go:117] "RemoveContainer" containerID="e36021cb33c2aafbfefd5d8b3b4fe44ebb838c8d94c95138a8464413bfa25619" Jul 06 19:09:11.981648 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:11.981632 2553 generic.go:356] "Generic (PLEG): container finished" podID="632d5de0-2e1b-4124-906e-21a096713b69" containerID="bb81e665dc41db2d20aa1ae59107d9dcc1eaf39606e5bedec62700c9a2a9de12" exitCode=1 Jul 06 19:09:11.981730 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:11.981704 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" event={"ID":"632d5de0-2e1b-4124-906e-21a096713b69","Type":"ContainerDied","Data":"bb81e665dc41db2d20aa1ae59107d9dcc1eaf39606e5bedec62700c9a2a9de12"} Jul 06 19:09:11.981952 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:11.981937 2553 scope.go:117] "RemoveContainer" containerID="bb81e665dc41db2d20aa1ae59107d9dcc1eaf39606e5bedec62700c9a2a9de12" Jul 06 19:09:11.983193 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:11.983175 2553 generic.go:356] "Generic (PLEG): container finished" podID="2ab81f98-e81a-4b24-beae-8babf214f1fd" containerID="deb554c8ea8e9d64e25a5590a2e44cf8ffa1ff75566ac7956d0185600c8d1264" exitCode=1 Jul 06 19:09:11.983269 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:11.983253 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" event={"ID":"2ab81f98-e81a-4b24-beae-8babf214f1fd","Type":"ContainerDied","Data":"deb554c8ea8e9d64e25a5590a2e44cf8ffa1ff75566ac7956d0185600c8d1264"} Jul 06 19:09:11.983540 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:11.983527 2553 scope.go:117] "RemoveContainer" containerID="deb554c8ea8e9d64e25a5590a2e44cf8ffa1ff75566ac7956d0185600c8d1264" Jul 06 19:09:12.402538 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:09:12.402448 2553 controller.go:195] "Failed to update lease" err="Put \"https://aa02cc370fcfb46c9bb69b2b56b189b2-b1b7f0f080e2fe4d.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-141-237.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jul 06 19:09:12.989215 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:12.989180 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" event={"ID":"ba75546a-9e3d-4363-a68d-b194380ff105","Type":"ContainerStarted","Data":"f668202ff17fbbb16bea784714365a5dd609df24a1e4bc180d6599dbe8619d0e"} Jul 06 19:09:12.989709 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:12.989399 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:09:12.991063 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:12.991041 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" event={"ID":"ad5da93d-e1e5-4854-98fb-7d8f95143b13","Type":"ContainerStarted","Data":"9a293a8a22e4123929f2441e2508eced98864e621ef276bb04c81c878f17df08"} Jul 06 19:09:12.991256 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:12.991242 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:09:12.992781 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:12.992753 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" event={"ID":"632d5de0-2e1b-4124-906e-21a096713b69","Type":"ContainerStarted","Data":"176030f8f3d1339bf1f1b1a562e964c7835c4fcee176dc22aca75ffce650ba65"} Jul 06 19:09:12.992936 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:12.992923 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" Jul 06 19:09:12.994457 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:12.994437 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" event={"ID":"2ab81f98-e81a-4b24-beae-8babf214f1fd","Type":"ContainerStarted","Data":"e5cb613a78c2d82cdf9eb9680a860c2e5f059e6e5e49e631f58548371ecfd4c1"} Jul 06 19:09:12.994618 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:12.994605 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" Jul 06 19:09:14.057776 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:09:14.057740 2553 controller.go:195] "Failed to update lease" err="Operation cannot be fulfilled on leases.coordination.k8s.io \"ip-10-0-141-237.ec2.internal\": the object has been modified; please apply your changes to the latest version and try again" Jul 06 19:09:15.036356 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.036323 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97"] Jul 06 19:09:15.036699 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.036687 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1aa414-a92b-4531-9ec9-5603bd025d65" containerName="manager" Jul 06 19:09:15.036755 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.036701 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1aa414-a92b-4531-9ec9-5603bd025d65" containerName="manager" Jul 06 19:09:15.036787 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.036770 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b1aa414-a92b-4531-9ec9-5603bd025d65" containerName="manager" Jul 06 19:09:15.038627 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.038612 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.051876 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.051856 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fg8r5\"" Jul 06 19:09:15.068795 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.068770 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b30dece-56c5-458a-82b9-f799d264cb5a-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.069109 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.068806 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b30dece-56c5-458a-82b9-f799d264cb5a-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.069109 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.068827 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b30dece-56c5-458a-82b9-f799d264cb5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.069109 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.068872 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b30dece-56c5-458a-82b9-f799d264cb5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.069109 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.068893 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b30dece-56c5-458a-82b9-f799d264cb5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.069109 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.068922 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b30dece-56c5-458a-82b9-f799d264cb5a-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.069109 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.069002 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b30dece-56c5-458a-82b9-f799d264cb5a-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.069109 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.069043 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b30dece-56c5-458a-82b9-f799d264cb5a-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.069109 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.069071 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7gs9\" (UniqueName: \"kubernetes.io/projected/6b30dece-56c5-458a-82b9-f799d264cb5a-kube-api-access-k7gs9\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.156986 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.156955 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97"] Jul 06 19:09:15.170436 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.170384 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b30dece-56c5-458a-82b9-f799d264cb5a-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.170573 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.170443 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b30dece-56c5-458a-82b9-f799d264cb5a-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.170573 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.170475 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b30dece-56c5-458a-82b9-f799d264cb5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.170573 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.170504 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b30dece-56c5-458a-82b9-f799d264cb5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.170573 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.170529 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b30dece-56c5-458a-82b9-f799d264cb5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.170573 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.170566 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b30dece-56c5-458a-82b9-f799d264cb5a-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.170838 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.170610 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b30dece-56c5-458a-82b9-f799d264cb5a-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.170838 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.170645 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b30dece-56c5-458a-82b9-f799d264cb5a-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.170838 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.170672 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7gs9\" (UniqueName: \"kubernetes.io/projected/6b30dece-56c5-458a-82b9-f799d264cb5a-kube-api-access-k7gs9\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.170838 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.170826 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b30dece-56c5-458a-82b9-f799d264cb5a-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.171045 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.170935 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b30dece-56c5-458a-82b9-f799d264cb5a-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.171102 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.171054 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b30dece-56c5-458a-82b9-f799d264cb5a-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.171102 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.171089 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b30dece-56c5-458a-82b9-f799d264cb5a-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.171837 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.171808 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b30dece-56c5-458a-82b9-f799d264cb5a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.172931 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.172901 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b30dece-56c5-458a-82b9-f799d264cb5a-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.173262 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.173244 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b30dece-56c5-458a-82b9-f799d264cb5a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.198013 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.197986 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7gs9\" (UniqueName: \"kubernetes.io/projected/6b30dece-56c5-458a-82b9-f799d264cb5a-kube-api-access-k7gs9\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.202594 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.202578 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b30dece-56c5-458a-82b9-f799d264cb5a-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-lcj97\" (UID: \"6b30dece-56c5-458a-82b9-f799d264cb5a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.348911 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.348830 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:15.499321 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.499288 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97"] Jul 06 19:09:15.501980 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:09:15.501953 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b30dece_56c5_458a_82b9_f799d264cb5a.slice/crio-0721499dc98ffe347d115de6e04b759d472a01e8bbf3189ffc31dfc356ef5ee7 WatchSource:0}: Error finding container 0721499dc98ffe347d115de6e04b759d472a01e8bbf3189ffc31dfc356ef5ee7: Status 404 returned error can't find the container with id 0721499dc98ffe347d115de6e04b759d472a01e8bbf3189ffc31dfc356ef5ee7 Jul 06 19:09:15.504134 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.504103 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:09:15.504217 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.504168 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:09:15.504217 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:15.504196 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:09:16.012311 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:16.012272 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" event={"ID":"6b30dece-56c5-458a-82b9-f799d264cb5a","Type":"ContainerStarted","Data":"3d3095fc8caec3aef7642a9bd395f7dd18461b0d07a4c84e4dab5d78b9514618"} Jul 06 19:09:16.012538 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:16.012325 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" event={"ID":"6b30dece-56c5-458a-82b9-f799d264cb5a","Type":"ContainerStarted","Data":"0721499dc98ffe347d115de6e04b759d472a01e8bbf3189ffc31dfc356ef5ee7"} Jul 06 19:09:16.067394 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:16.067344 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" podStartSLOduration=1.0673276409999999 podStartE2EDuration="1.067327641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:09:16.065365094 +0000 UTC m=+617.334547449" watchObservedRunningTime="2026-07-06 19:09:16.067327641 +0000 UTC m=+617.336509982" Jul 06 19:09:16.349551 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:16.349461 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:16.354251 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:16.354225 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:17.017473 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:17.017444 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:17.018544 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:17.018525 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:09:24.000834 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:24.000758 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" Jul 06 19:09:24.000834 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:24.000829 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" Jul 06 19:09:34.101505 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:34.101467 2553 generic.go:356] "Generic (PLEG): container finished" podID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerID="8f17ee150552d55727ac712237cc31455f8a8d253c3a5f4a4c47d59c6f0d91a6" exitCode=0 Jul 06 19:09:34.101895 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:34.101518 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" event={"ID":"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413","Type":"ContainerDied","Data":"8f17ee150552d55727ac712237cc31455f8a8d253c3a5f4a4c47d59c6f0d91a6"} Jul 06 19:09:34.101895 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:34.101796 2553 scope.go:117] "RemoveContainer" containerID="8f17ee150552d55727ac712237cc31455f8a8d253c3a5f4a4c47d59c6f0d91a6" Jul 06 19:09:34.496902 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:09:34.496860 2553 controller.go:195] "Failed to update lease" err="Put \"https://aa02cc370fcfb46c9bb69b2b56b189b2-b1b7f0f080e2fe4d.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-141-237.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jul 06 19:09:35.108013 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:35.107974 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" event={"ID":"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413","Type":"ContainerStarted","Data":"a026a74afbad0e6b811492471921e9bb5f59c042e87b0fb92fc3ac59c1493be7"} Jul 06 19:09:35.108494 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:35.108216 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:09:35.412219 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:09:35.412130 2553 controller.go:195] "Failed to update lease" err="Operation cannot be fulfilled on leases.coordination.k8s.io \"ip-10-0-141-237.ec2.internal\": the object has been modified; please apply your changes to the latest version and try again" Jul 06 19:09:44.001364 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:44.001330 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:09:44.004657 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:44.004636 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:09:46.119220 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:46.119188 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:09:54.030477 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:54.030448 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:09:54.035114 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:54.035079 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-xh6vt" Jul 06 19:09:54.042613 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:54.042587 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 19:09:54.067720 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:54.067691 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:09:54.103282 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:54.103255 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qls9q\" (UniqueName: \"kubernetes.io/projected/18559278-46e1-4a51-a067-8a11b7414264-kube-api-access-qls9q\") pod \"limitador-limitador-54d6d988d5-xh6vt\" (UID: \"18559278-46e1-4a51-a067-8a11b7414264\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xh6vt" Jul 06 19:09:54.103392 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:54.103292 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/18559278-46e1-4a51-a067-8a11b7414264-config-file\") pod \"limitador-limitador-54d6d988d5-xh6vt\" (UID: \"18559278-46e1-4a51-a067-8a11b7414264\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xh6vt" Jul 06 19:09:54.127505 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:54.127480 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:09:54.204334 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:54.204310 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qls9q\" (UniqueName: \"kubernetes.io/projected/18559278-46e1-4a51-a067-8a11b7414264-kube-api-access-qls9q\") pod \"limitador-limitador-54d6d988d5-xh6vt\" (UID: \"18559278-46e1-4a51-a067-8a11b7414264\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xh6vt" Jul 06 19:09:54.204510 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:54.204344 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/18559278-46e1-4a51-a067-8a11b7414264-config-file\") pod \"limitador-limitador-54d6d988d5-xh6vt\" (UID: \"18559278-46e1-4a51-a067-8a11b7414264\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xh6vt" Jul 06 19:09:54.204934 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:54.204916 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/18559278-46e1-4a51-a067-8a11b7414264-config-file\") pod \"limitador-limitador-54d6d988d5-xh6vt\" (UID: \"18559278-46e1-4a51-a067-8a11b7414264\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xh6vt" Jul 06 19:09:54.216945 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:54.216918 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qls9q\" (UniqueName: \"kubernetes.io/projected/18559278-46e1-4a51-a067-8a11b7414264-kube-api-access-qls9q\") pod \"limitador-limitador-54d6d988d5-xh6vt\" (UID: \"18559278-46e1-4a51-a067-8a11b7414264\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xh6vt" Jul 06 19:09:54.346536 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:54.346463 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-xh6vt" Jul 06 19:09:54.486722 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:54.486697 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:09:54.489069 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:09:54.489044 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18559278_46e1_4a51_a067_8a11b7414264.slice/crio-65746d9149c7d540b6ceba2d660defa4b400a4657af422fbb9487e5906032ac1 WatchSource:0}: Error finding container 65746d9149c7d540b6ceba2d660defa4b400a4657af422fbb9487e5906032ac1: Status 404 returned error can't find the container with id 65746d9149c7d540b6ceba2d660defa4b400a4657af422fbb9487e5906032ac1 Jul 06 19:09:55.208891 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:55.208847 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-xh6vt" event={"ID":"18559278-46e1-4a51-a067-8a11b7414264","Type":"ContainerStarted","Data":"65746d9149c7d540b6ceba2d660defa4b400a4657af422fbb9487e5906032ac1"} Jul 06 19:09:58.225039 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:58.225008 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-xh6vt" event={"ID":"18559278-46e1-4a51-a067-8a11b7414264","Type":"ContainerStarted","Data":"f0b67a3f83266f4c33cd7464ae09f69f5516d92eb9d743d1b3a42470a7665416"} Jul 06 19:09:58.225406 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:58.225124 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-xh6vt" Jul 06 19:09:58.253140 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:09:58.253100 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-xh6vt" podStartSLOduration=2.237425779 podStartE2EDuration="5.253087012s" totalImagesPullingTime="3.015661233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:09:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:09:58.250917425 +0000 UTC m=+659.520099777" watchObservedRunningTime="2026-07-06 19:09:58.253087012 +0000 UTC m=+659.522269354" Jul 06 19:10:09.230410 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:09.230377 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-xh6vt" Jul 06 19:10:43.999285 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:43.999243 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" podUID="2ab81f98-e81a-4b24-beae-8babf214f1fd" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.32:8081/readyz\": dial tcp 10.132.0.32:8081: connect: connection refused" Jul 06 19:10:43.999755 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:43.999242 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" podUID="632d5de0-2e1b-4124-906e-21a096713b69" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.29:8081/readyz\": dial tcp 10.132.0.29:8081: connect: connection refused" Jul 06 19:10:43.999755 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:43.999243 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" podUID="ad5da93d-e1e5-4854-98fb-7d8f95143b13" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.25:8081/readyz\": dial tcp 10.132.0.25:8081: connect: connection refused" Jul 06 19:10:43.999755 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:43.999243 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" podUID="ba75546a-9e3d-4363-a68d-b194380ff105" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.26:8081/readyz\": dial tcp 10.132.0.26:8081: connect: connection refused" Jul 06 19:10:44.441314 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.441236 2553 generic.go:356] "Generic (PLEG): container finished" podID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerID="a026a74afbad0e6b811492471921e9bb5f59c042e87b0fb92fc3ac59c1493be7" exitCode=0 Jul 06 19:10:44.441500 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.441311 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" event={"ID":"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413","Type":"ContainerDied","Data":"a026a74afbad0e6b811492471921e9bb5f59c042e87b0fb92fc3ac59c1493be7"} Jul 06 19:10:44.441500 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.441382 2553 scope.go:117] "RemoveContainer" containerID="8f17ee150552d55727ac712237cc31455f8a8d253c3a5f4a4c47d59c6f0d91a6" Jul 06 19:10:44.441772 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.441750 2553 scope.go:117] "RemoveContainer" containerID="a026a74afbad0e6b811492471921e9bb5f59c042e87b0fb92fc3ac59c1493be7" Jul 06 19:10:44.441998 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:10:44.441975 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kuadrant-operator-controller-manager-86878c769f-khb2s_kuadrant-system(f889fc14-3f1f-4bf4-ba6d-3f8e5f804413)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" Jul 06 19:10:44.443406 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.443388 2553 generic.go:356] "Generic (PLEG): container finished" podID="ba75546a-9e3d-4363-a68d-b194380ff105" containerID="f668202ff17fbbb16bea784714365a5dd609df24a1e4bc180d6599dbe8619d0e" exitCode=1 Jul 06 19:10:44.443536 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.443448 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" event={"ID":"ba75546a-9e3d-4363-a68d-b194380ff105","Type":"ContainerDied","Data":"f668202ff17fbbb16bea784714365a5dd609df24a1e4bc180d6599dbe8619d0e"} Jul 06 19:10:44.443779 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.443763 2553 scope.go:117] "RemoveContainer" containerID="f668202ff17fbbb16bea784714365a5dd609df24a1e4bc180d6599dbe8619d0e" Jul 06 19:10:44.443970 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:10:44.443950 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-sq8dz_opendatahub(ba75546a-9e3d-4363-a68d-b194380ff105)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" podUID="ba75546a-9e3d-4363-a68d-b194380ff105" Jul 06 19:10:44.445157 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.445137 2553 generic.go:356] "Generic (PLEG): container finished" podID="ad5da93d-e1e5-4854-98fb-7d8f95143b13" containerID="9a293a8a22e4123929f2441e2508eced98864e621ef276bb04c81c878f17df08" exitCode=1 Jul 06 19:10:44.445251 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.445206 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" event={"ID":"ad5da93d-e1e5-4854-98fb-7d8f95143b13","Type":"ContainerDied","Data":"9a293a8a22e4123929f2441e2508eced98864e621ef276bb04c81c878f17df08"} Jul 06 19:10:44.445568 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.445554 2553 scope.go:117] "RemoveContainer" containerID="9a293a8a22e4123929f2441e2508eced98864e621ef276bb04c81c878f17df08" Jul 06 19:10:44.445741 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:10:44.445722 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-gwbxn_opendatahub(ad5da93d-e1e5-4854-98fb-7d8f95143b13)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" podUID="ad5da93d-e1e5-4854-98fb-7d8f95143b13" Jul 06 19:10:44.447136 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.447117 2553 generic.go:356] "Generic (PLEG): container finished" podID="632d5de0-2e1b-4124-906e-21a096713b69" containerID="176030f8f3d1339bf1f1b1a562e964c7835c4fcee176dc22aca75ffce650ba65" exitCode=1 Jul 06 19:10:44.447229 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.447186 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" event={"ID":"632d5de0-2e1b-4124-906e-21a096713b69","Type":"ContainerDied","Data":"176030f8f3d1339bf1f1b1a562e964c7835c4fcee176dc22aca75ffce650ba65"} Jul 06 19:10:44.447520 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.447500 2553 scope.go:117] "RemoveContainer" containerID="176030f8f3d1339bf1f1b1a562e964c7835c4fcee176dc22aca75ffce650ba65" Jul 06 19:10:44.447710 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:10:44.447676 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=dns-operator-controller-manager-864dd78c4c-wklwt_kuadrant-system(632d5de0-2e1b-4124-906e-21a096713b69)\"" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" podUID="632d5de0-2e1b-4124-906e-21a096713b69" Jul 06 19:10:44.449071 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.449052 2553 generic.go:356] "Generic (PLEG): container finished" podID="2ab81f98-e81a-4b24-beae-8babf214f1fd" containerID="e5cb613a78c2d82cdf9eb9680a860c2e5f059e6e5e49e631f58548371ecfd4c1" exitCode=1 Jul 06 19:10:44.449167 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.449124 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" event={"ID":"2ab81f98-e81a-4b24-beae-8babf214f1fd","Type":"ContainerDied","Data":"e5cb613a78c2d82cdf9eb9680a860c2e5f059e6e5e49e631f58548371ecfd4c1"} Jul 06 19:10:44.449402 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.449387 2553 scope.go:117] "RemoveContainer" containerID="e5cb613a78c2d82cdf9eb9680a860c2e5f059e6e5e49e631f58548371ecfd4c1" Jul 06 19:10:44.449581 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:10:44.449563 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=authorino-operator-75cc8d5cb7-kr5j9_kuadrant-system(2ab81f98-e81a-4b24-beae-8babf214f1fd)\"" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" podUID="2ab81f98-e81a-4b24-beae-8babf214f1fd" Jul 06 19:10:44.452243 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.452228 2553 scope.go:117] "RemoveContainer" containerID="1b021fb3585448a2f827c409396d923b786991270bbebae9e776c807e826efe0" Jul 06 19:10:44.462651 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.462633 2553 scope.go:117] "RemoveContainer" containerID="e36021cb33c2aafbfefd5d8b3b4fe44ebb838c8d94c95138a8464413bfa25619" Jul 06 19:10:44.471724 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.471708 2553 scope.go:117] "RemoveContainer" containerID="bb81e665dc41db2d20aa1ae59107d9dcc1eaf39606e5bedec62700c9a2a9de12" Jul 06 19:10:44.480514 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:44.480496 2553 scope.go:117] "RemoveContainer" containerID="deb554c8ea8e9d64e25a5590a2e44cf8ffa1ff75566ac7956d0185600c8d1264" Jul 06 19:10:46.117875 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:46.117835 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:10:46.118251 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:46.118143 2553 scope.go:117] "RemoveContainer" containerID="a026a74afbad0e6b811492471921e9bb5f59c042e87b0fb92fc3ac59c1493be7" Jul 06 19:10:46.118340 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:10:46.118320 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kuadrant-operator-controller-manager-86878c769f-khb2s_kuadrant-system(f889fc14-3f1f-4bf4-ba6d-3f8e5f804413)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" Jul 06 19:10:48.707072 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:48.707031 2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:10:48.707481 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:48.707391 2553 scope.go:117] "RemoveContainer" containerID="9a293a8a22e4123929f2441e2508eced98864e621ef276bb04c81c878f17df08" Jul 06 19:10:48.707611 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:10:48.707591 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-gwbxn_opendatahub(ad5da93d-e1e5-4854-98fb-7d8f95143b13)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" podUID="ad5da93d-e1e5-4854-98fb-7d8f95143b13" Jul 06 19:10:48.712494 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:48.712462 2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:10:48.712773 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:48.712761 2553 scope.go:117] "RemoveContainer" containerID="f668202ff17fbbb16bea784714365a5dd609df24a1e4bc180d6599dbe8619d0e" Jul 06 19:10:48.712927 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:10:48.712911 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-sq8dz_opendatahub(ba75546a-9e3d-4363-a68d-b194380ff105)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" podUID="ba75546a-9e3d-4363-a68d-b194380ff105" Jul 06 19:10:48.999995 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:48.999965 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:10:49.000159 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:49.000021 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:10:49.478008 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:49.477983 2553 scope.go:117] "RemoveContainer" containerID="9a293a8a22e4123929f2441e2508eced98864e621ef276bb04c81c878f17df08" Jul 06 19:10:49.478151 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:49.478087 2553 scope.go:117] "RemoveContainer" containerID="f668202ff17fbbb16bea784714365a5dd609df24a1e4bc180d6599dbe8619d0e" Jul 06 19:10:49.478218 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:10:49.478196 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-gwbxn_opendatahub(ad5da93d-e1e5-4854-98fb-7d8f95143b13)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" podUID="ad5da93d-e1e5-4854-98fb-7d8f95143b13" Jul 06 19:10:49.478274 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:10:49.478257 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-sq8dz_opendatahub(ba75546a-9e3d-4363-a68d-b194380ff105)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" podUID="ba75546a-9e3d-4363-a68d-b194380ff105" Jul 06 19:10:49.742045 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:10:49.741950 2553 controller.go:195] "Failed to update lease" err="Put \"https://aa02cc370fcfb46c9bb69b2b56b189b2-b1b7f0f080e2fe4d.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-141-237.ec2.internal?timeout=10s\": context deadline exceeded" Jul 06 19:10:50.157568 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:50.157491 2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" Jul 06 19:10:50.157857 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:50.157843 2553 scope.go:117] "RemoveContainer" containerID="176030f8f3d1339bf1f1b1a562e964c7835c4fcee176dc22aca75ffce650ba65" Jul 06 19:10:50.158028 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:10:50.158011 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=dns-operator-controller-manager-864dd78c4c-wklwt_kuadrant-system(632d5de0-2e1b-4124-906e-21a096713b69)\"" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" podUID="632d5de0-2e1b-4124-906e-21a096713b69" Jul 06 19:10:53.999992 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:53.999961 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" Jul 06 19:10:54.000354 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:54.000017 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" Jul 06 19:10:54.000354 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:54.000291 2553 scope.go:117] "RemoveContainer" containerID="176030f8f3d1339bf1f1b1a562e964c7835c4fcee176dc22aca75ffce650ba65" Jul 06 19:10:54.000502 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:54.000486 2553 scope.go:117] "RemoveContainer" containerID="e5cb613a78c2d82cdf9eb9680a860c2e5f059e6e5e49e631f58548371ecfd4c1" Jul 06 19:10:54.501744 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:54.501713 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" event={"ID":"632d5de0-2e1b-4124-906e-21a096713b69","Type":"ContainerStarted","Data":"f9683fc339300c283cb0914414cc917eca82dc5c06f5b5e99eddfee7b98433a7"} Jul 06 19:10:54.502001 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:54.501964 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" Jul 06 19:10:54.503570 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:54.503552 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" event={"ID":"2ab81f98-e81a-4b24-beae-8babf214f1fd","Type":"ContainerStarted","Data":"77cab6887db8ff177b90b3c3e3a4e7234e097b6ff5c3c160ab80532b0ff992a8"} Jul 06 19:10:54.503748 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:54.503733 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" Jul 06 19:10:58.276960 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:58.276925 2553 scope.go:117] "RemoveContainer" containerID="a026a74afbad0e6b811492471921e9bb5f59c042e87b0fb92fc3ac59c1493be7" Jul 06 19:10:58.522595 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:58.522565 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" event={"ID":"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413","Type":"ContainerStarted","Data":"f376da8a4730a8a455d0bb69aeb5a4b8e7f788a9580e71d5789a822af923cca9"} Jul 06 19:10:58.522795 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:10:58.522777 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:10:59.742648 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:10:59.742614 2553 controller.go:195] "Failed to update lease" err="Put \"https://aa02cc370fcfb46c9bb69b2b56b189b2-b1b7f0f080e2fe4d.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-141-237.ec2.internal?timeout=10s\": context deadline exceeded" Jul 06 19:11:01.276841 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:01.276810 2553 scope.go:117] "RemoveContainer" containerID="9a293a8a22e4123929f2441e2508eced98864e621ef276bb04c81c878f17df08" Jul 06 19:11:01.277200 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:01.277075 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:11:02.543519 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:02.543483 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" event={"ID":"ad5da93d-e1e5-4854-98fb-7d8f95143b13","Type":"ContainerStarted","Data":"23c4af7242e444922f3fab23c146729f1d957010118958274435e384e0e909e1"} Jul 06 19:11:02.543927 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:02.543700 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:11:03.276844 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:03.276810 2553 scope.go:117] "RemoveContainer" containerID="f668202ff17fbbb16bea784714365a5dd609df24a1e4bc180d6599dbe8619d0e" Jul 06 19:11:04.559872 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:04.559835 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" event={"ID":"ba75546a-9e3d-4363-a68d-b194380ff105","Type":"ContainerStarted","Data":"50d4e2130061c8259c6d09c444fe50f19b06958d3e43056fa0986e6899b8c5a9"} Jul 06 19:11:04.560335 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:04.560092 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:11:04.561675 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:04.561654 2553 generic.go:356] "Generic (PLEG): container finished" podID="632d5de0-2e1b-4124-906e-21a096713b69" containerID="f9683fc339300c283cb0914414cc917eca82dc5c06f5b5e99eddfee7b98433a7" exitCode=1 Jul 06 19:11:04.561774 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:04.561695 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" event={"ID":"632d5de0-2e1b-4124-906e-21a096713b69","Type":"ContainerDied","Data":"f9683fc339300c283cb0914414cc917eca82dc5c06f5b5e99eddfee7b98433a7"} Jul 06 19:11:04.561774 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:04.561729 2553 scope.go:117] "RemoveContainer" containerID="176030f8f3d1339bf1f1b1a562e964c7835c4fcee176dc22aca75ffce650ba65" Jul 06 19:11:04.561970 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:04.561953 2553 scope.go:117] "RemoveContainer" containerID="f9683fc339300c283cb0914414cc917eca82dc5c06f5b5e99eddfee7b98433a7" Jul 06 19:11:04.562186 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:11:04.562167 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=dns-operator-controller-manager-864dd78c4c-wklwt_kuadrant-system(632d5de0-2e1b-4124-906e-21a096713b69)\"" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" podUID="632d5de0-2e1b-4124-906e-21a096713b69" Jul 06 19:11:05.510533 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:05.510502 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" Jul 06 19:11:08.583460 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:08.583433 2553 generic.go:356] "Generic (PLEG): container finished" podID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerID="f376da8a4730a8a455d0bb69aeb5a4b8e7f788a9580e71d5789a822af923cca9" exitCode=1 Jul 06 19:11:08.583857 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:08.583477 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" event={"ID":"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413","Type":"ContainerDied","Data":"f376da8a4730a8a455d0bb69aeb5a4b8e7f788a9580e71d5789a822af923cca9"} Jul 06 19:11:08.583857 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:08.583542 2553 scope.go:117] "RemoveContainer" containerID="a026a74afbad0e6b811492471921e9bb5f59c042e87b0fb92fc3ac59c1493be7" Jul 06 19:11:08.583980 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:08.583893 2553 scope.go:117] "RemoveContainer" containerID="f376da8a4730a8a455d0bb69aeb5a4b8e7f788a9580e71d5789a822af923cca9" Jul 06 19:11:08.584120 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:11:08.584100 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=kuadrant-operator-controller-manager-86878c769f-khb2s_kuadrant-system(f889fc14-3f1f-4bf4-ba6d-3f8e5f804413)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" Jul 06 19:11:09.743079 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:11:09.743047 2553 controller.go:195] "Failed to update lease" err="Put \"https://aa02cc370fcfb46c9bb69b2b56b189b2-b1b7f0f080e2fe4d.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-141-237.ec2.internal?timeout=10s\": context deadline exceeded" Jul 06 19:11:10.157179 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:10.157106 2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" Jul 06 19:11:10.157402 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:10.157389 2553 scope.go:117] "RemoveContainer" containerID="f9683fc339300c283cb0914414cc917eca82dc5c06f5b5e99eddfee7b98433a7" Jul 06 19:11:10.157605 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:11:10.157585 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=dns-operator-controller-manager-864dd78c4c-wklwt_kuadrant-system(632d5de0-2e1b-4124-906e-21a096713b69)\"" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" podUID="632d5de0-2e1b-4124-906e-21a096713b69" Jul 06 19:11:12.604821 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:12.604786 2553 generic.go:356] "Generic (PLEG): container finished" podID="ad5da93d-e1e5-4854-98fb-7d8f95143b13" containerID="23c4af7242e444922f3fab23c146729f1d957010118958274435e384e0e909e1" exitCode=1 Jul 06 19:11:12.605261 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:12.604859 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" event={"ID":"ad5da93d-e1e5-4854-98fb-7d8f95143b13","Type":"ContainerDied","Data":"23c4af7242e444922f3fab23c146729f1d957010118958274435e384e0e909e1"} Jul 06 19:11:12.605261 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:12.604936 2553 scope.go:117] "RemoveContainer" containerID="9a293a8a22e4123929f2441e2508eced98864e621ef276bb04c81c878f17df08" Jul 06 19:11:12.605347 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:12.605291 2553 scope.go:117] "RemoveContainer" containerID="23c4af7242e444922f3fab23c146729f1d957010118958274435e384e0e909e1" Jul 06 19:11:12.605562 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:11:12.605541 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-gwbxn_opendatahub(ad5da93d-e1e5-4854-98fb-7d8f95143b13)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" podUID="ad5da93d-e1e5-4854-98fb-7d8f95143b13" Jul 06 19:11:14.620441 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:14.620337 2553 generic.go:356] "Generic (PLEG): container finished" podID="ba75546a-9e3d-4363-a68d-b194380ff105" containerID="50d4e2130061c8259c6d09c444fe50f19b06958d3e43056fa0986e6899b8c5a9" exitCode=1 Jul 06 19:11:14.620441 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:14.620385 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" event={"ID":"ba75546a-9e3d-4363-a68d-b194380ff105","Type":"ContainerDied","Data":"50d4e2130061c8259c6d09c444fe50f19b06958d3e43056fa0986e6899b8c5a9"} Jul 06 19:11:14.620987 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:14.620472 2553 scope.go:117] "RemoveContainer" containerID="f668202ff17fbbb16bea784714365a5dd609df24a1e4bc180d6599dbe8619d0e" Jul 06 19:11:14.620987 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:14.620878 2553 scope.go:117] "RemoveContainer" containerID="50d4e2130061c8259c6d09c444fe50f19b06958d3e43056fa0986e6899b8c5a9" Jul 06 19:11:14.621083 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:11:14.621057 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-sq8dz_opendatahub(ba75546a-9e3d-4363-a68d-b194380ff105)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" podUID="ba75546a-9e3d-4363-a68d-b194380ff105" Jul 06 19:11:18.707135 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:18.707091 2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:11:18.707564 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:18.707472 2553 scope.go:117] "RemoveContainer" containerID="23c4af7242e444922f3fab23c146729f1d957010118958274435e384e0e909e1" Jul 06 19:11:18.707651 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:11:18.707632 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-gwbxn_opendatahub(ad5da93d-e1e5-4854-98fb-7d8f95143b13)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" podUID="ad5da93d-e1e5-4854-98fb-7d8f95143b13" Jul 06 19:11:18.712534 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:18.712512 2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:11:18.712820 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:18.712805 2553 scope.go:117] "RemoveContainer" containerID="50d4e2130061c8259c6d09c444fe50f19b06958d3e43056fa0986e6899b8c5a9" Jul 06 19:11:18.713010 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:11:18.712990 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-sq8dz_opendatahub(ba75546a-9e3d-4363-a68d-b194380ff105)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" podUID="ba75546a-9e3d-4363-a68d-b194380ff105" Jul 06 19:11:19.744216 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:11:19.744180 2553 controller.go:195] "Failed to update lease" err="Put \"https://aa02cc370fcfb46c9bb69b2b56b189b2-b1b7f0f080e2fe4d.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-141-237.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jul 06 19:11:21.276849 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:21.276822 2553 scope.go:117] "RemoveContainer" containerID="f376da8a4730a8a455d0bb69aeb5a4b8e7f788a9580e71d5789a822af923cca9" Jul 06 19:11:21.277202 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:11:21.276975 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=kuadrant-operator-controller-manager-86878c769f-khb2s_kuadrant-system(f889fc14-3f1f-4bf4-ba6d-3f8e5f804413)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" Jul 06 19:11:22.606122 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:22.606094 2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:11:22.606539 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:22.606462 2553 scope.go:117] "RemoveContainer" containerID="f376da8a4730a8a455d0bb69aeb5a4b8e7f788a9580e71d5789a822af923cca9" Jul 06 19:11:22.606657 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:11:22.606638 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=kuadrant-operator-controller-manager-86878c769f-khb2s_kuadrant-system(f889fc14-3f1f-4bf4-ba6d-3f8e5f804413)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" Jul 06 19:11:23.277248 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:23.277218 2553 scope.go:117] "RemoveContainer" containerID="f9683fc339300c283cb0914414cc917eca82dc5c06f5b5e99eddfee7b98433a7" Jul 06 19:11:23.277496 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:11:23.277472 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=dns-operator-controller-manager-864dd78c4c-wklwt_kuadrant-system(632d5de0-2e1b-4124-906e-21a096713b69)\"" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" podUID="632d5de0-2e1b-4124-906e-21a096713b69" Jul 06 19:11:30.277736 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:30.277701 2553 scope.go:117] "RemoveContainer" containerID="23c4af7242e444922f3fab23c146729f1d957010118958274435e384e0e909e1" Jul 06 19:11:30.278166 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:11:30.277930 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-gwbxn_opendatahub(ad5da93d-e1e5-4854-98fb-7d8f95143b13)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" podUID="ad5da93d-e1e5-4854-98fb-7d8f95143b13" Jul 06 19:11:33.283123 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:33.283097 2553 scope.go:117] "RemoveContainer" containerID="50d4e2130061c8259c6d09c444fe50f19b06958d3e43056fa0986e6899b8c5a9" Jul 06 19:11:33.283496 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:11:33.283265 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-sq8dz_opendatahub(ba75546a-9e3d-4363-a68d-b194380ff105)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" podUID="ba75546a-9e3d-4363-a68d-b194380ff105" Jul 06 19:11:34.277660 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:34.277633 2553 scope.go:117] "RemoveContainer" containerID="f9683fc339300c283cb0914414cc917eca82dc5c06f5b5e99eddfee7b98433a7" Jul 06 19:11:34.714582 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:34.714502 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" event={"ID":"632d5de0-2e1b-4124-906e-21a096713b69","Type":"ContainerStarted","Data":"65b2a0ff8b8bffecce0462ebb85c58f56373e015420cf152870eb7fea724251a"} Jul 06 19:11:34.714927 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:34.714712 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" Jul 06 19:11:37.079757 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:11:37.079711 2553 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-07-06T19:11:27Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-06T19:11:27Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-06T19:11:27Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-07-06T19:11:27Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-141-237.ec2.internal\": Patch \"https://aa02cc370fcfb46c9bb69b2b56b189b2-b1b7f0f080e2fe4d.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-141-237.ec2.internal/status?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jul 06 19:11:37.283128 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:37.283101 2553 scope.go:117] "RemoveContainer" containerID="f376da8a4730a8a455d0bb69aeb5a4b8e7f788a9580e71d5789a822af923cca9" Jul 06 19:11:37.729617 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:37.729584 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" event={"ID":"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413","Type":"ContainerStarted","Data":"f7daf9eca56db79d2a29a8388660184279e59db0ade798452c83a37cfa8bbd12"} Jul 06 19:11:37.729827 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:37.729809 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:11:45.277088 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:45.277052 2553 scope.go:117] "RemoveContainer" containerID="23c4af7242e444922f3fab23c146729f1d957010118958274435e384e0e909e1" Jul 06 19:11:45.277488 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:45.277103 2553 scope.go:117] "RemoveContainer" containerID="50d4e2130061c8259c6d09c444fe50f19b06958d3e43056fa0986e6899b8c5a9" Jul 06 19:11:45.720729 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:45.720707 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" Jul 06 19:11:45.768225 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:45.768200 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" event={"ID":"ad5da93d-e1e5-4854-98fb-7d8f95143b13","Type":"ContainerStarted","Data":"84ae5a59adc865fa0dca22291a2c31444e4944930870517ee750d33adc7d23df"} Jul 06 19:11:45.768395 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:45.768382 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:11:46.775334 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:46.775294 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" event={"ID":"ba75546a-9e3d-4363-a68d-b194380ff105","Type":"ContainerStarted","Data":"7f43cf2817f92a1573650d3487f893d9751c6be1ea56bc18f1513c7289c6a0ad"} Jul 06 19:11:46.775813 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:46.775574 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:11:48.736093 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:48.736067 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:11:50.795322 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:50.795296 2553 generic.go:356] "Generic (PLEG): container finished" podID="ad5da93d-e1e5-4854-98fb-7d8f95143b13" containerID="84ae5a59adc865fa0dca22291a2c31444e4944930870517ee750d33adc7d23df" exitCode=1 Jul 06 19:11:50.795622 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:50.795367 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" event={"ID":"ad5da93d-e1e5-4854-98fb-7d8f95143b13","Type":"ContainerDied","Data":"84ae5a59adc865fa0dca22291a2c31444e4944930870517ee750d33adc7d23df"} Jul 06 19:11:50.795622 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:50.795493 2553 scope.go:117] "RemoveContainer" containerID="23c4af7242e444922f3fab23c146729f1d957010118958274435e384e0e909e1" Jul 06 19:11:50.795786 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:50.795771 2553 scope.go:117] "RemoveContainer" containerID="84ae5a59adc865fa0dca22291a2c31444e4944930870517ee750d33adc7d23df" Jul 06 19:11:50.796005 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:11:50.795982 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-gwbxn_opendatahub(ad5da93d-e1e5-4854-98fb-7d8f95143b13)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" podUID="ad5da93d-e1e5-4854-98fb-7d8f95143b13" Jul 06 19:11:53.291104 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:11:53.291053 2553 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413\\\"},\\\"status\\\":{\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://f7daf9eca56db79d2a29a8388660184279e59db0ade798452c83a37cfa8bbd12\\\",\\\"image\\\":\\\"quay.io/kuadrant/kuadrant-operator:v1.4.2\\\",\\\"imageID\\\":\\\"quay.io/kuadrant/kuadrant-operator@sha256:0ad6673d1a66a99ce9881d4f50b51e1f98b477a019a8efe7187a73396d773e58\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://f376da8a4730a8a455d0bb69aeb5a4b8e7f788a9580e71d5789a822af923cca9\\\",\\\"exitCode\\\":1,\\\"finishedAt\\\":\\\"2026-07-06T19:11:08Z\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-07-06T19:10:58Z\\\"}},\\\"name\\\":\\\"manager\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"300Mi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":3,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-06T19:11:37Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000680000],\\\"uid\\\":1000680000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/tmp/kuadrant\\\",\\\"name\\\":\\\"extensions-socket-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-kbx78\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-86878c769f-khb2s\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-ed6006ff-308f-4519-b84f-e305d9418cee.svc:9743/pod?timeout=10s\": context deadline exceeded" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:11:58.706438 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:58.706383 2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:11:58.706826 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:11:58.706770 2553 scope.go:117] "RemoveContainer" containerID="84ae5a59adc865fa0dca22291a2c31444e4944930870517ee750d33adc7d23df" Jul 06 19:11:58.706985 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:11:58.706966 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-gwbxn_opendatahub(ad5da93d-e1e5-4854-98fb-7d8f95143b13)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" podUID="ad5da93d-e1e5-4854-98fb-7d8f95143b13" Jul 06 19:12:03.318865 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:12:03.318811 2553 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"ad5da93d-e1e5-4854-98fb-7d8f95143b13\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-06T19:10:44Z\\\",\\\"message\\\":\\\"containers with unready status: [manager]\\\",\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-07-06T19:10:44Z\\\",\\\"message\\\":\\\"containers with unready status: [manager]\\\",\\\"reason\\\":\\\"ContainersNotReady\\\",\\\"status\\\":\\\"False\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://23c4af7242e444922f3fab23c146729f1d957010118958274435e384e0e909e1\\\",\\\"image\\\":\\\"quay.io/opendatahub/kserve-controller:odh-v3.5-EA2\\\",\\\"imageID\\\":\\\"quay.io/opendatahub/kserve-controller@sha256:0367496c5c5155e53df50ad25a64a7016e5f7496ae22a4c9dedce7d296b8e9e1\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://23c4af7242e444922f3fab23c146729f1d957010118958274435e384e0e909e1\\\",\\\"exitCode\\\":1,\\\"finishedAt\\\":\\\"2026-07-06T19:11:11Z\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-07-06T19:11:01Z\\\"}},\\\"name\\\":\\\"manager\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"500m\\\",\\\"memory\\\":\\\"500Mi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"100m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":2,\\\"started\\\":false,\\\"state\\\":{\\\"waiting\\\":{\\\"message\\\":\\\"back-off 20s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-gwbxn_opendatahub(ad5da93d-e1e5-4854-98fb-7d8f95143b13)\\\",\\\"reason\\\":\\\"CrashLoopBackOff\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000660000],\\\"uid\\\":1000660000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/tmp/k8s-webhook-server/serving-certs\\\",\\\"name\\\":\\\"cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-qc5cn\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"opendatahub\"/\"kserve-controller-manager-d6f4b65b8-gwbxn\": Internal error occurred: failed calling webhook \"pod.network-node-identity.openshift.io\": failed to call webhook: Post \"https://network-node-identity.clusters-ed6006ff-308f-4519-b84f-e305d9418cee.svc:9743/pod?timeout=10s\": context deadline exceeded" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:12:13.277059 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:13.277028 2553 scope.go:117] "RemoveContainer" containerID="84ae5a59adc865fa0dca22291a2c31444e4944930870517ee750d33adc7d23df" Jul 06 19:12:13.277447 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:12:13.277237 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-gwbxn_opendatahub(ad5da93d-e1e5-4854-98fb-7d8f95143b13)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" podUID="ad5da93d-e1e5-4854-98fb-7d8f95143b13" Jul 06 19:12:17.783182 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:17.783153 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:12:27.277765 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:27.277694 2553 scope.go:117] "RemoveContainer" containerID="84ae5a59adc865fa0dca22291a2c31444e4944930870517ee750d33adc7d23df" Jul 06 19:12:27.278124 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:12:27.277904 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-gwbxn_opendatahub(ad5da93d-e1e5-4854-98fb-7d8f95143b13)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" podUID="ad5da93d-e1e5-4854-98fb-7d8f95143b13" Jul 06 19:12:30.157242 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:30.157207 2553 prober.go:120] "Probe failed" probeType="Liveness" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" podUID="632d5de0-2e1b-4124-906e-21a096713b69" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.29:8081/healthz\": dial tcp 10.132.0.29:8081: connect: connection refused" Jul 06 19:12:30.984483 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:30.984445 2553 generic.go:356] "Generic (PLEG): container finished" podID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerID="f7daf9eca56db79d2a29a8388660184279e59db0ade798452c83a37cfa8bbd12" exitCode=0 Jul 06 19:12:30.984663 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:30.984511 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" event={"ID":"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413","Type":"ContainerDied","Data":"f7daf9eca56db79d2a29a8388660184279e59db0ade798452c83a37cfa8bbd12"} Jul 06 19:12:30.984663 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:30.984568 2553 scope.go:117] "RemoveContainer" containerID="f376da8a4730a8a455d0bb69aeb5a4b8e7f788a9580e71d5789a822af923cca9" Jul 06 19:12:30.984863 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:30.984848 2553 scope.go:117] "RemoveContainer" containerID="f7daf9eca56db79d2a29a8388660184279e59db0ade798452c83a37cfa8bbd12" Jul 06 19:12:30.985045 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:12:30.985022 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=kuadrant-operator-controller-manager-86878c769f-khb2s_kuadrant-system(f889fc14-3f1f-4bf4-ba6d-3f8e5f804413)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" Jul 06 19:12:30.986578 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:30.986559 2553 generic.go:356] "Generic (PLEG): container finished" podID="ba75546a-9e3d-4363-a68d-b194380ff105" containerID="7f43cf2817f92a1573650d3487f893d9751c6be1ea56bc18f1513c7289c6a0ad" exitCode=1 Jul 06 19:12:30.986715 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:30.986630 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" event={"ID":"ba75546a-9e3d-4363-a68d-b194380ff105","Type":"ContainerDied","Data":"7f43cf2817f92a1573650d3487f893d9751c6be1ea56bc18f1513c7289c6a0ad"} Jul 06 19:12:30.986920 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:30.986907 2553 scope.go:117] "RemoveContainer" containerID="7f43cf2817f92a1573650d3487f893d9751c6be1ea56bc18f1513c7289c6a0ad" Jul 06 19:12:30.987101 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:12:30.987086 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-sq8dz_opendatahub(ba75546a-9e3d-4363-a68d-b194380ff105)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" podUID="ba75546a-9e3d-4363-a68d-b194380ff105" Jul 06 19:12:30.988462 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:30.988438 2553 generic.go:356] "Generic (PLEG): container finished" podID="632d5de0-2e1b-4124-906e-21a096713b69" containerID="65b2a0ff8b8bffecce0462ebb85c58f56373e015420cf152870eb7fea724251a" exitCode=1 Jul 06 19:12:30.988583 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:30.988498 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" event={"ID":"632d5de0-2e1b-4124-906e-21a096713b69","Type":"ContainerDied","Data":"65b2a0ff8b8bffecce0462ebb85c58f56373e015420cf152870eb7fea724251a"} Jul 06 19:12:30.988765 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:30.988750 2553 scope.go:117] "RemoveContainer" containerID="65b2a0ff8b8bffecce0462ebb85c58f56373e015420cf152870eb7fea724251a" Jul 06 19:12:30.988911 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:12:30.988891 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=dns-operator-controller-manager-864dd78c4c-wklwt_kuadrant-system(632d5de0-2e1b-4124-906e-21a096713b69)\"" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" podUID="632d5de0-2e1b-4124-906e-21a096713b69" Jul 06 19:12:30.994709 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:30.994687 2553 scope.go:117] "RemoveContainer" containerID="50d4e2130061c8259c6d09c444fe50f19b06958d3e43056fa0986e6899b8c5a9" Jul 06 19:12:31.006284 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:31.006269 2553 scope.go:117] "RemoveContainer" containerID="f9683fc339300c283cb0914414cc917eca82dc5c06f5b5e99eddfee7b98433a7" Jul 06 19:12:32.780951 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:32.780917 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:12:32.781305 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:32.781287 2553 scope.go:117] "RemoveContainer" containerID="7f43cf2817f92a1573650d3487f893d9751c6be1ea56bc18f1513c7289c6a0ad" Jul 06 19:12:32.781488 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:12:32.781470 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-sq8dz_opendatahub(ba75546a-9e3d-4363-a68d-b194380ff105)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" podUID="ba75546a-9e3d-4363-a68d-b194380ff105" Jul 06 19:12:35.719834 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:35.719792 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" Jul 06 19:12:35.720283 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:35.720147 2553 scope.go:117] "RemoveContainer" containerID="65b2a0ff8b8bffecce0462ebb85c58f56373e015420cf152870eb7fea724251a" Jul 06 19:12:35.720341 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:12:35.720320 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=dns-operator-controller-manager-864dd78c4c-wklwt_kuadrant-system(632d5de0-2e1b-4124-906e-21a096713b69)\"" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" podUID="632d5de0-2e1b-4124-906e-21a096713b69" Jul 06 19:12:38.277487 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:38.277456 2553 scope.go:117] "RemoveContainer" containerID="84ae5a59adc865fa0dca22291a2c31444e4944930870517ee750d33adc7d23df" Jul 06 19:12:38.712004 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:38.711931 2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:12:38.712304 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:38.712292 2553 scope.go:117] "RemoveContainer" containerID="7f43cf2817f92a1573650d3487f893d9751c6be1ea56bc18f1513c7289c6a0ad" Jul 06 19:12:38.712508 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:12:38.712489 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-sq8dz_opendatahub(ba75546a-9e3d-4363-a68d-b194380ff105)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" podUID="ba75546a-9e3d-4363-a68d-b194380ff105" Jul 06 19:12:38.733894 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:38.733873 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:12:38.734215 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:38.734200 2553 scope.go:117] "RemoveContainer" containerID="f7daf9eca56db79d2a29a8388660184279e59db0ade798452c83a37cfa8bbd12" Jul 06 19:12:38.734395 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:12:38.734376 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=kuadrant-operator-controller-manager-86878c769f-khb2s_kuadrant-system(f889fc14-3f1f-4bf4-ba6d-3f8e5f804413)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" Jul 06 19:12:39.032150 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:39.032122 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" event={"ID":"ad5da93d-e1e5-4854-98fb-7d8f95143b13","Type":"ContainerStarted","Data":"9e018db6d17018afdbd958b5060e2fe5e5c65d99f56d4fc31239ed7e70e76b4a"} Jul 06 19:12:39.032350 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:39.032335 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:12:42.606524 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:42.606485 2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:12:42.606996 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:42.606970 2553 scope.go:117] "RemoveContainer" containerID="f7daf9eca56db79d2a29a8388660184279e59db0ade798452c83a37cfa8bbd12" Jul 06 19:12:42.607197 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:12:42.607173 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=kuadrant-operator-controller-manager-86878c769f-khb2s_kuadrant-system(f889fc14-3f1f-4bf4-ba6d-3f8e5f804413)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" Jul 06 19:12:48.277479 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:48.277448 2553 scope.go:117] "RemoveContainer" containerID="65b2a0ff8b8bffecce0462ebb85c58f56373e015420cf152870eb7fea724251a" Jul 06 19:12:48.277926 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:12:48.277628 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=dns-operator-controller-manager-864dd78c4c-wklwt_kuadrant-system(632d5de0-2e1b-4124-906e-21a096713b69)\"" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" podUID="632d5de0-2e1b-4124-906e-21a096713b69" Jul 06 19:12:50.157176 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:50.157146 2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" Jul 06 19:12:50.157559 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:50.157505 2553 scope.go:117] "RemoveContainer" containerID="65b2a0ff8b8bffecce0462ebb85c58f56373e015420cf152870eb7fea724251a" Jul 06 19:12:50.157680 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:12:50.157660 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=dns-operator-controller-manager-864dd78c4c-wklwt_kuadrant-system(632d5de0-2e1b-4124-906e-21a096713b69)\"" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" podUID="632d5de0-2e1b-4124-906e-21a096713b69" Jul 06 19:12:54.277211 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:54.277176 2553 scope.go:117] "RemoveContainer" containerID="7f43cf2817f92a1573650d3487f893d9751c6be1ea56bc18f1513c7289c6a0ad" Jul 06 19:12:54.277619 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:12:54.277323 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-sq8dz_opendatahub(ba75546a-9e3d-4363-a68d-b194380ff105)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" podUID="ba75546a-9e3d-4363-a68d-b194380ff105" Jul 06 19:12:56.277281 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:12:56.277250 2553 scope.go:117] "RemoveContainer" containerID="f7daf9eca56db79d2a29a8388660184279e59db0ade798452c83a37cfa8bbd12" Jul 06 19:12:56.277717 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:12:56.277439 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=kuadrant-operator-controller-manager-86878c769f-khb2s_kuadrant-system(f889fc14-3f1f-4bf4-ba6d-3f8e5f804413)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" Jul 06 19:13:04.277351 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:04.277322 2553 scope.go:117] "RemoveContainer" containerID="65b2a0ff8b8bffecce0462ebb85c58f56373e015420cf152870eb7fea724251a" Jul 06 19:13:04.277762 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:13:04.277533 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=dns-operator-controller-manager-864dd78c4c-wklwt_kuadrant-system(632d5de0-2e1b-4124-906e-21a096713b69)\"" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" podUID="632d5de0-2e1b-4124-906e-21a096713b69" Jul 06 19:13:08.277308 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:08.277280 2553 scope.go:117] "RemoveContainer" containerID="7f43cf2817f92a1573650d3487f893d9751c6be1ea56bc18f1513c7289c6a0ad" Jul 06 19:13:08.277697 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:13:08.277473 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-sq8dz_opendatahub(ba75546a-9e3d-4363-a68d-b194380ff105)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" podUID="ba75546a-9e3d-4363-a68d-b194380ff105" Jul 06 19:13:10.040749 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:10.040720 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwbxn" Jul 06 19:13:11.282616 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:11.282588 2553 scope.go:117] "RemoveContainer" containerID="f7daf9eca56db79d2a29a8388660184279e59db0ade798452c83a37cfa8bbd12" Jul 06 19:13:12.189486 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:12.189452 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" event={"ID":"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413","Type":"ContainerStarted","Data":"001ae6fba608f47c776e68c5ddcc9bef3a2850458d00276d416bf8bb050f1cec"} Jul 06 19:13:12.189668 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:12.189652 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:13:15.276749 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:15.276718 2553 scope.go:117] "RemoveContainer" containerID="65b2a0ff8b8bffecce0462ebb85c58f56373e015420cf152870eb7fea724251a" Jul 06 19:13:16.212109 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:16.212077 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" event={"ID":"632d5de0-2e1b-4124-906e-21a096713b69","Type":"ContainerStarted","Data":"59e3ff88a9519763d242a3affe7c0991916f64ed7cb6fbf69ea38e081ade5103"} Jul 06 19:13:16.212295 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:16.212275 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" Jul 06 19:13:19.279136 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:19.279111 2553 scope.go:117] "RemoveContainer" containerID="7f43cf2817f92a1573650d3487f893d9751c6be1ea56bc18f1513c7289c6a0ad" Jul 06 19:13:20.230880 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:20.230845 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" event={"ID":"ba75546a-9e3d-4363-a68d-b194380ff105","Type":"ContainerStarted","Data":"ddec642faf61dbe69ebc92cff1774ac792a170316dca08beb75f77990fe01e75"} Jul 06 19:13:20.231075 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:20.231057 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:13:23.197900 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:23.197870 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:13:27.218188 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:27.218156 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wklwt" Jul 06 19:13:34.593555 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:34.593521 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:13:38.314740 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:38.314700 2553 generic.go:356] "Generic (PLEG): container finished" podID="6b30dece-56c5-458a-82b9-f799d264cb5a" containerID="3d3095fc8caec3aef7642a9bd395f7dd18461b0d07a4c84e4dab5d78b9514618" exitCode=0 Jul 06 19:13:38.315171 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:38.314781 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" event={"ID":"6b30dece-56c5-458a-82b9-f799d264cb5a","Type":"ContainerDied","Data":"3d3095fc8caec3aef7642a9bd395f7dd18461b0d07a4c84e4dab5d78b9514618"} Jul 06 19:13:38.315171 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:38.315142 2553 scope.go:117] "RemoveContainer" containerID="3d3095fc8caec3aef7642a9bd395f7dd18461b0d07a4c84e4dab5d78b9514618" Jul 06 19:13:38.316100 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:38.316074 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:13:38.316158 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:38.316146 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:13:38.316206 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:38.316194 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 19:13:39.324075 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:39.324034 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" event={"ID":"6b30dece-56c5-458a-82b9-f799d264cb5a","Type":"ContainerStarted","Data":"53c9f7e575a167ff98c472c6a01627327a5afd4d51f4c80dac2221b1abbe125f"} Jul 06 19:13:39.349664 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:39.349629 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:13:39.349858 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:39.349831 2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lcj97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 06 19:13:39.349913 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:39.349894 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" podUID="6b30dece-56c5-458a-82b9-f799d264cb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 06 19:13:40.349471 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:40.349437 2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lcj97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 06 19:13:40.349817 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:40.349485 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" podUID="6b30dece-56c5-458a-82b9-f799d264cb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 06 19:13:41.349907 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:41.349874 2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lcj97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 06 19:13:41.350301 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:41.349926 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" podUID="6b30dece-56c5-458a-82b9-f799d264cb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 06 19:13:42.349328 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:42.349295 2553 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-lcj97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Jul 06 19:13:42.349519 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:42.349355 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" podUID="6b30dece-56c5-458a-82b9-f799d264cb5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Jul 06 19:13:43.352665 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:43.352638 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:13:43.353053 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:43.352901 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:13:43.353620 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:43.353605 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-lcj97" Jul 06 19:13:51.238393 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:51.238318 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sq8dz" Jul 06 19:13:59.190659 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:59.190631 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:13:59.193466 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:59.193440 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:13:59.196748 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:59.196726 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:13:59.199087 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:13:59.199068 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:14:48.777603 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:14:48.777571 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:15:04.903709 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:04.903665 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722755-55kz6"] Jul 06 19:15:04.906149 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:04.906118 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722755-55kz6" Jul 06 19:15:04.908346 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:04.908327 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-zxgdj\"" Jul 06 19:15:04.915737 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:04.915710 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722755-55kz6"] Jul 06 19:15:04.965526 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:04.965495 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h85d\" (UniqueName: \"kubernetes.io/projected/d82ab975-83d9-4c4b-93ee-13c52fced6b3-kube-api-access-8h85d\") pod \"maas-api-key-cleanup-29722755-55kz6\" (UID: \"d82ab975-83d9-4c4b-93ee-13c52fced6b3\") " pod="opendatahub/maas-api-key-cleanup-29722755-55kz6" Jul 06 19:15:05.066016 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:05.065978 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h85d\" (UniqueName: \"kubernetes.io/projected/d82ab975-83d9-4c4b-93ee-13c52fced6b3-kube-api-access-8h85d\") pod \"maas-api-key-cleanup-29722755-55kz6\" (UID: \"d82ab975-83d9-4c4b-93ee-13c52fced6b3\") " pod="opendatahub/maas-api-key-cleanup-29722755-55kz6" Jul 06 19:15:05.076831 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:05.076804 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h85d\" (UniqueName: \"kubernetes.io/projected/d82ab975-83d9-4c4b-93ee-13c52fced6b3-kube-api-access-8h85d\") pod \"maas-api-key-cleanup-29722755-55kz6\" (UID: \"d82ab975-83d9-4c4b-93ee-13c52fced6b3\") " pod="opendatahub/maas-api-key-cleanup-29722755-55kz6" Jul 06 19:15:05.217025 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:05.216945 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722755-55kz6" Jul 06 19:15:05.351083 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:05.351051 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722755-55kz6"] Jul 06 19:15:05.354374 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:15:05.354348 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82ab975_83d9_4c4b_93ee_13c52fced6b3.slice/crio-4c3fc2bc4b04700253556e54a46c30dde9fca867d375018ec9d74f02d6bf205b WatchSource:0}: Error finding container 4c3fc2bc4b04700253556e54a46c30dde9fca867d375018ec9d74f02d6bf205b: Status 404 returned error can't find the container with id 4c3fc2bc4b04700253556e54a46c30dde9fca867d375018ec9d74f02d6bf205b Jul 06 19:15:05.727736 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:05.727701 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722755-55kz6" event={"ID":"d82ab975-83d9-4c4b-93ee-13c52fced6b3","Type":"ContainerStarted","Data":"4c3fc2bc4b04700253556e54a46c30dde9fca867d375018ec9d74f02d6bf205b"} Jul 06 19:15:07.740234 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:07.740202 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722755-55kz6" event={"ID":"d82ab975-83d9-4c4b-93ee-13c52fced6b3","Type":"ContainerStarted","Data":"9e3bfccbee2e35a3aec5386020b0187f21cbf89d001d9699346d8eb39e47d228"} Jul 06 19:15:08.745832 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:08.745800 2553 generic.go:356] "Generic (PLEG): container finished" podID="d82ab975-83d9-4c4b-93ee-13c52fced6b3" containerID="9e3bfccbee2e35a3aec5386020b0187f21cbf89d001d9699346d8eb39e47d228" exitCode=0 Jul 06 19:15:08.746299 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:08.745885 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722755-55kz6" event={"ID":"d82ab975-83d9-4c4b-93ee-13c52fced6b3","Type":"ContainerDied","Data":"9e3bfccbee2e35a3aec5386020b0187f21cbf89d001d9699346d8eb39e47d228"} Jul 06 19:15:09.878270 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:09.878245 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722755-55kz6" Jul 06 19:15:09.909587 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:09.909557 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8h85d\" (UniqueName: \"kubernetes.io/projected/d82ab975-83d9-4c4b-93ee-13c52fced6b3-kube-api-access-8h85d\") pod \"d82ab975-83d9-4c4b-93ee-13c52fced6b3\" (UID: \"d82ab975-83d9-4c4b-93ee-13c52fced6b3\") " Jul 06 19:15:09.911856 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:09.911832 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d82ab975-83d9-4c4b-93ee-13c52fced6b3-kube-api-access-8h85d" (OuterVolumeSpecName: "kube-api-access-8h85d") pod "d82ab975-83d9-4c4b-93ee-13c52fced6b3" (UID: "d82ab975-83d9-4c4b-93ee-13c52fced6b3"). InnerVolumeSpecName "kube-api-access-8h85d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:15:09.946333 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:09.946282 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722755-55kz6" podStartSLOduration=4.040844307 podStartE2EDuration="5.946262301s" totalImagesPullingTime="1.905417994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:15:09.945998424 +0000 UTC m=+971.215180813" watchObservedRunningTime="2026-07-06 19:15:09.946262301 +0000 UTC m=+971.215444645" Jul 06 19:15:10.010846 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:10.010776 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h85d\" (UniqueName: \"kubernetes.io/projected/d82ab975-83d9-4c4b-93ee-13c52fced6b3-kube-api-access-8h85d\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:15:10.755870 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:10.755832 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722755-55kz6" event={"ID":"d82ab975-83d9-4c4b-93ee-13c52fced6b3","Type":"ContainerDied","Data":"4c3fc2bc4b04700253556e54a46c30dde9fca867d375018ec9d74f02d6bf205b"} Jul 06 19:15:10.755870 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:10.755859 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722755-55kz6" Jul 06 19:15:10.756074 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:10.755893 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c3fc2bc4b04700253556e54a46c30dde9fca867d375018ec9d74f02d6bf205b" Jul 06 19:15:34.404346 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:34.404261 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:15:36.662489 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.662457 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj"] Jul 06 19:15:36.662837 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.662783 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d82ab975-83d9-4c4b-93ee-13c52fced6b3" containerName="cleanup" Jul 06 19:15:36.662837 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.662793 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d82ab975-83d9-4c4b-93ee-13c52fced6b3" containerName="cleanup" Jul 06 19:15:36.662907 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.662867 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d82ab975-83d9-4c4b-93ee-13c52fced6b3" containerName="cleanup" Jul 06 19:15:36.665834 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.665818 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:36.668271 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.668247 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kppft\"" Jul 06 19:15:36.669828 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.669810 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 06 19:15:36.670040 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.670027 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 19:15:36.670269 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.670256 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 19:15:36.693670 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.693649 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj"] Jul 06 19:15:36.846401 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.846367 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66840435-d9c5-4e2c-9890-c62a74662880-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q7njj\" (UID: \"66840435-d9c5-4e2c-9890-c62a74662880\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:36.846593 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.846440 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66840435-d9c5-4e2c-9890-c62a74662880-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q7njj\" (UID: \"66840435-d9c5-4e2c-9890-c62a74662880\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:36.846593 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.846513 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66840435-d9c5-4e2c-9890-c62a74662880-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q7njj\" (UID: \"66840435-d9c5-4e2c-9890-c62a74662880\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:36.846593 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.846580 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66840435-d9c5-4e2c-9890-c62a74662880-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q7njj\" (UID: \"66840435-d9c5-4e2c-9890-c62a74662880\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:36.846731 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.846603 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66840435-d9c5-4e2c-9890-c62a74662880-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q7njj\" (UID: \"66840435-d9c5-4e2c-9890-c62a74662880\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:36.846731 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.846629 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x89r7\" (UniqueName: \"kubernetes.io/projected/66840435-d9c5-4e2c-9890-c62a74662880-kube-api-access-x89r7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q7njj\" (UID: \"66840435-d9c5-4e2c-9890-c62a74662880\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:36.948108 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.948016 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66840435-d9c5-4e2c-9890-c62a74662880-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q7njj\" (UID: \"66840435-d9c5-4e2c-9890-c62a74662880\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:36.948108 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.948065 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x89r7\" (UniqueName: \"kubernetes.io/projected/66840435-d9c5-4e2c-9890-c62a74662880-kube-api-access-x89r7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q7njj\" (UID: \"66840435-d9c5-4e2c-9890-c62a74662880\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:36.948108 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.948099 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66840435-d9c5-4e2c-9890-c62a74662880-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q7njj\" (UID: \"66840435-d9c5-4e2c-9890-c62a74662880\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:36.948380 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.948148 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66840435-d9c5-4e2c-9890-c62a74662880-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q7njj\" (UID: \"66840435-d9c5-4e2c-9890-c62a74662880\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:36.948380 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.948185 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66840435-d9c5-4e2c-9890-c62a74662880-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q7njj\" (UID: \"66840435-d9c5-4e2c-9890-c62a74662880\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:36.948380 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.948249 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66840435-d9c5-4e2c-9890-c62a74662880-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q7njj\" (UID: \"66840435-d9c5-4e2c-9890-c62a74662880\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:36.948569 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.948497 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66840435-d9c5-4e2c-9890-c62a74662880-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q7njj\" (UID: \"66840435-d9c5-4e2c-9890-c62a74662880\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:36.948632 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.948613 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66840435-d9c5-4e2c-9890-c62a74662880-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q7njj\" (UID: \"66840435-d9c5-4e2c-9890-c62a74662880\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:36.948678 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.948666 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66840435-d9c5-4e2c-9890-c62a74662880-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q7njj\" (UID: \"66840435-d9c5-4e2c-9890-c62a74662880\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:36.950517 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.950498 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66840435-d9c5-4e2c-9890-c62a74662880-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q7njj\" (UID: \"66840435-d9c5-4e2c-9890-c62a74662880\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:36.950845 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.950825 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66840435-d9c5-4e2c-9890-c62a74662880-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q7njj\" (UID: \"66840435-d9c5-4e2c-9890-c62a74662880\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:36.958702 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.958677 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x89r7\" (UniqueName: \"kubernetes.io/projected/66840435-d9c5-4e2c-9890-c62a74662880-kube-api-access-x89r7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-q7njj\" (UID: \"66840435-d9c5-4e2c-9890-c62a74662880\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:36.976496 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:36.976478 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:37.135832 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:37.135686 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj"] Jul 06 19:15:37.138538 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:15:37.138511 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66840435_d9c5_4e2c_9890_c62a74662880.slice/crio-317fb8cdffa1a927550d78d14d8f3c2606831eaf717c19e0675baa3dba5a25c5 WatchSource:0}: Error finding container 317fb8cdffa1a927550d78d14d8f3c2606831eaf717c19e0675baa3dba5a25c5: Status 404 returned error can't find the container with id 317fb8cdffa1a927550d78d14d8f3c2606831eaf717c19e0675baa3dba5a25c5 Jul 06 19:15:37.886325 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:37.886288 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" event={"ID":"66840435-d9c5-4e2c-9890-c62a74662880","Type":"ContainerStarted","Data":"317fb8cdffa1a927550d78d14d8f3c2606831eaf717c19e0675baa3dba5a25c5"} Jul 06 19:15:38.892839 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:38.892803 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" event={"ID":"66840435-d9c5-4e2c-9890-c62a74662880","Type":"ContainerStarted","Data":"2e0904fb97a25fcf5018a6f07cb370218d01715b6718c13470bb6b08e5488c86"} Jul 06 19:15:40.058472 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:40.058397 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" podStartSLOduration=2.988997524 podStartE2EDuration="4.058381902s" totalImagesPullingTime="1.069384378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:15:40.05519282 +0000 UTC m=+1001.324375164" watchObservedRunningTime="2026-07-06 19:15:40.058381902 +0000 UTC m=+1001.327564244" Jul 06 19:15:46.976846 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:46.976809 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:46.977226 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:46.976895 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:46.989505 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:46.989483 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:47.955371 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:47.955339 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-q7njj" Jul 06 19:15:57.527291 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:15:57.527254 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:16:07.766868 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:07.766828 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx"] Jul 06 19:16:07.770626 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:07.770607 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:07.785902 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:16:07.785877 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" type="*v1.Secret" Jul 06 19:16:07.785902 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:16:07.785886 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"llm\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="a76e45ae-863a-4358-985d-2295f154dd25" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:07.806216 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:07.806193 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx"] Jul 06 19:16:07.915878 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:07.915848 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76e45ae-863a-4358-985d-2295f154dd25-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx\" (UID: \"a76e45ae-863a-4358-985d-2295f154dd25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:07.916068 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:07.915896 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5dmk\" (UniqueName: \"kubernetes.io/projected/a76e45ae-863a-4358-985d-2295f154dd25-kube-api-access-m5dmk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx\" (UID: \"a76e45ae-863a-4358-985d-2295f154dd25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:07.916068 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:07.915928 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76e45ae-863a-4358-985d-2295f154dd25-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx\" (UID: \"a76e45ae-863a-4358-985d-2295f154dd25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:07.916068 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:07.916004 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76e45ae-863a-4358-985d-2295f154dd25-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx\" (UID: \"a76e45ae-863a-4358-985d-2295f154dd25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:07.916068 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:07.916048 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76e45ae-863a-4358-985d-2295f154dd25-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx\" (UID: \"a76e45ae-863a-4358-985d-2295f154dd25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:07.916068 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:07.916070 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76e45ae-863a-4358-985d-2295f154dd25-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx\" (UID: \"a76e45ae-863a-4358-985d-2295f154dd25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:08.016808 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:08.016778 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76e45ae-863a-4358-985d-2295f154dd25-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx\" (UID: \"a76e45ae-863a-4358-985d-2295f154dd25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:08.017025 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:08.016822 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76e45ae-863a-4358-985d-2295f154dd25-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx\" (UID: \"a76e45ae-863a-4358-985d-2295f154dd25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:08.017025 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:08.016842 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76e45ae-863a-4358-985d-2295f154dd25-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx\" (UID: \"a76e45ae-863a-4358-985d-2295f154dd25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:08.017025 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:08.016888 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76e45ae-863a-4358-985d-2295f154dd25-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx\" (UID: \"a76e45ae-863a-4358-985d-2295f154dd25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:08.017025 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:08.016910 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5dmk\" (UniqueName: \"kubernetes.io/projected/a76e45ae-863a-4358-985d-2295f154dd25-kube-api-access-m5dmk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx\" (UID: \"a76e45ae-863a-4358-985d-2295f154dd25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:08.017025 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:08.016928 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76e45ae-863a-4358-985d-2295f154dd25-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx\" (UID: \"a76e45ae-863a-4358-985d-2295f154dd25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:08.017302 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:08.017243 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76e45ae-863a-4358-985d-2295f154dd25-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx\" (UID: \"a76e45ae-863a-4358-985d-2295f154dd25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:08.017302 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:08.017280 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76e45ae-863a-4358-985d-2295f154dd25-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx\" (UID: \"a76e45ae-863a-4358-985d-2295f154dd25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:08.017302 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:08.017293 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76e45ae-863a-4358-985d-2295f154dd25-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx\" (UID: \"a76e45ae-863a-4358-985d-2295f154dd25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:08.019397 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:08.019379 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76e45ae-863a-4358-985d-2295f154dd25-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx\" (UID: \"a76e45ae-863a-4358-985d-2295f154dd25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:08.038529 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:08.038503 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5dmk\" (UniqueName: \"kubernetes.io/projected/a76e45ae-863a-4358-985d-2295f154dd25-kube-api-access-m5dmk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx\" (UID: \"a76e45ae-863a-4358-985d-2295f154dd25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:08.666155 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:08.666128 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 06 19:16:08.669632 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:08.669611 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76e45ae-863a-4358-985d-2295f154dd25-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx\" (UID: \"a76e45ae-863a-4358-985d-2295f154dd25\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:08.680652 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:08.680626 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:08.830132 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:08.830093 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx"] Jul 06 19:16:08.833004 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:16:08.832978 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76e45ae_863a_4358_985d_2295f154dd25.slice/crio-b57f9bad50613fa4fdb179853fecf7aa4fb2537a5150f12355bcbd4a9f39c9f3 WatchSource:0}: Error finding container b57f9bad50613fa4fdb179853fecf7aa4fb2537a5150f12355bcbd4a9f39c9f3: Status 404 returned error can't find the container with id b57f9bad50613fa4fdb179853fecf7aa4fb2537a5150f12355bcbd4a9f39c9f3 Jul 06 19:16:08.834253 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:08.834235 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:16:09.044551 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:09.044520 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" event={"ID":"a76e45ae-863a-4358-985d-2295f154dd25","Type":"ContainerStarted","Data":"b57f9bad50613fa4fdb179853fecf7aa4fb2537a5150f12355bcbd4a9f39c9f3"} Jul 06 19:16:09.766601 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:09.766568 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:16:10.051402 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:10.051316 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" event={"ID":"a76e45ae-863a-4358-985d-2295f154dd25","Type":"ContainerStarted","Data":"35f26ff0d6932b2dea0807e725a2ff7d7a022ac83e65d10a00548b0a38257712"} Jul 06 19:16:10.140652 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:10.140597 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" podStartSLOduration=2.897566851 podStartE2EDuration="3.140578107s" totalImagesPullingTime="243.011256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:10.136160031 +0000 UTC m=+1031.405342374" watchObservedRunningTime="2026-07-06 19:16:10.140578107 +0000 UTC m=+1031.409760451" Jul 06 19:16:12.387266 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.387229 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t"] Jul 06 19:16:12.392394 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.392375 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.400568 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.400550 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 06 19:16:12.461671 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.461640 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d60fae4e-c9ac-4baf-88ad-647922d26bd3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpm2t\" (UID: \"d60fae4e-c9ac-4baf-88ad-647922d26bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.461789 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.461676 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d60fae4e-c9ac-4baf-88ad-647922d26bd3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpm2t\" (UID: \"d60fae4e-c9ac-4baf-88ad-647922d26bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.461789 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.461702 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d60fae4e-c9ac-4baf-88ad-647922d26bd3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpm2t\" (UID: \"d60fae4e-c9ac-4baf-88ad-647922d26bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.461867 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.461815 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d60fae4e-c9ac-4baf-88ad-647922d26bd3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpm2t\" (UID: \"d60fae4e-c9ac-4baf-88ad-647922d26bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.461867 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.461851 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw4pt\" (UniqueName: \"kubernetes.io/projected/d60fae4e-c9ac-4baf-88ad-647922d26bd3-kube-api-access-zw4pt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpm2t\" (UID: \"d60fae4e-c9ac-4baf-88ad-647922d26bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.461930 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.461883 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d60fae4e-c9ac-4baf-88ad-647922d26bd3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpm2t\" (UID: \"d60fae4e-c9ac-4baf-88ad-647922d26bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.501236 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.501205 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t"] Jul 06 19:16:12.562449 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.562400 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw4pt\" (UniqueName: \"kubernetes.io/projected/d60fae4e-c9ac-4baf-88ad-647922d26bd3-kube-api-access-zw4pt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpm2t\" (UID: \"d60fae4e-c9ac-4baf-88ad-647922d26bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.562615 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.562461 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d60fae4e-c9ac-4baf-88ad-647922d26bd3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpm2t\" (UID: \"d60fae4e-c9ac-4baf-88ad-647922d26bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.562615 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.562505 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d60fae4e-c9ac-4baf-88ad-647922d26bd3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpm2t\" (UID: \"d60fae4e-c9ac-4baf-88ad-647922d26bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.562615 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.562527 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d60fae4e-c9ac-4baf-88ad-647922d26bd3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpm2t\" (UID: \"d60fae4e-c9ac-4baf-88ad-647922d26bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.562615 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.562545 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d60fae4e-c9ac-4baf-88ad-647922d26bd3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpm2t\" (UID: \"d60fae4e-c9ac-4baf-88ad-647922d26bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.562615 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.562612 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d60fae4e-c9ac-4baf-88ad-647922d26bd3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpm2t\" (UID: \"d60fae4e-c9ac-4baf-88ad-647922d26bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.563000 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.562971 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d60fae4e-c9ac-4baf-88ad-647922d26bd3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpm2t\" (UID: \"d60fae4e-c9ac-4baf-88ad-647922d26bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.563115 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.563043 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d60fae4e-c9ac-4baf-88ad-647922d26bd3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpm2t\" (UID: \"d60fae4e-c9ac-4baf-88ad-647922d26bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.563115 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.563067 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d60fae4e-c9ac-4baf-88ad-647922d26bd3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpm2t\" (UID: \"d60fae4e-c9ac-4baf-88ad-647922d26bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.565042 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.565015 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d60fae4e-c9ac-4baf-88ad-647922d26bd3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpm2t\" (UID: \"d60fae4e-c9ac-4baf-88ad-647922d26bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.565469 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.565447 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d60fae4e-c9ac-4baf-88ad-647922d26bd3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpm2t\" (UID: \"d60fae4e-c9ac-4baf-88ad-647922d26bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.669029 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.668961 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw4pt\" (UniqueName: \"kubernetes.io/projected/d60fae4e-c9ac-4baf-88ad-647922d26bd3-kube-api-access-zw4pt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-mpm2t\" (UID: \"d60fae4e-c9ac-4baf-88ad-647922d26bd3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.703141 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.703107 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:12.857493 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:12.857451 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t"] Jul 06 19:16:12.861566 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:16:12.861538 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60fae4e_c9ac_4baf_88ad_647922d26bd3.slice/crio-f3c2c3d68c093fde098180aed2dff67182894a79726ba325366efab82f3ca9e9 WatchSource:0}: Error finding container f3c2c3d68c093fde098180aed2dff67182894a79726ba325366efab82f3ca9e9: Status 404 returned error can't find the container with id f3c2c3d68c093fde098180aed2dff67182894a79726ba325366efab82f3ca9e9 Jul 06 19:16:13.067470 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:13.067439 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" event={"ID":"d60fae4e-c9ac-4baf-88ad-647922d26bd3","Type":"ContainerStarted","Data":"f3c2c3d68c093fde098180aed2dff67182894a79726ba325366efab82f3ca9e9"} Jul 06 19:16:14.073222 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:14.073192 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" event={"ID":"d60fae4e-c9ac-4baf-88ad-647922d26bd3","Type":"ContainerStarted","Data":"2cd820e3c3cc67a172d5a1100c5e8e793284c5963bb0dc5c8aec48e42f567d0e"} Jul 06 19:16:14.118389 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:14.118342 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" podStartSLOduration=1.9141383730000001 podStartE2EDuration="2.118327349s" totalImagesPullingTime="204.188976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:14.116947992 +0000 UTC m=+1035.386130336" watchObservedRunningTime="2026-07-06 19:16:14.118327349 +0000 UTC m=+1035.387509691" Jul 06 19:16:15.892371 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:15.892337 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:16:18.681397 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:18.681359 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:18.681843 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:18.681442 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:18.693593 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:18.693572 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:19.109446 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:19.109398 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx" Jul 06 19:16:21.348752 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.348718 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw"] Jul 06 19:16:21.351558 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.351541 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.360019 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.360001 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 06 19:16:21.392846 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.392819 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw"] Jul 06 19:16:21.443531 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.443501 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80b76a2-b565-4c5f-b234-0db0d4ded27f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw\" (UID: \"f80b76a2-b565-4c5f-b234-0db0d4ded27f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.443716 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.443546 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80b76a2-b565-4c5f-b234-0db0d4ded27f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw\" (UID: \"f80b76a2-b565-4c5f-b234-0db0d4ded27f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.443716 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.443571 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z8p5\" (UniqueName: \"kubernetes.io/projected/f80b76a2-b565-4c5f-b234-0db0d4ded27f-kube-api-access-6z8p5\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw\" (UID: \"f80b76a2-b565-4c5f-b234-0db0d4ded27f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.443716 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.443657 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80b76a2-b565-4c5f-b234-0db0d4ded27f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw\" (UID: \"f80b76a2-b565-4c5f-b234-0db0d4ded27f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.443843 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.443728 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80b76a2-b565-4c5f-b234-0db0d4ded27f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw\" (UID: \"f80b76a2-b565-4c5f-b234-0db0d4ded27f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.443843 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.443823 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80b76a2-b565-4c5f-b234-0db0d4ded27f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw\" (UID: \"f80b76a2-b565-4c5f-b234-0db0d4ded27f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.545137 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.545099 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80b76a2-b565-4c5f-b234-0db0d4ded27f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw\" (UID: \"f80b76a2-b565-4c5f-b234-0db0d4ded27f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.545330 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.545155 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80b76a2-b565-4c5f-b234-0db0d4ded27f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw\" (UID: \"f80b76a2-b565-4c5f-b234-0db0d4ded27f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.545330 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.545187 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z8p5\" (UniqueName: \"kubernetes.io/projected/f80b76a2-b565-4c5f-b234-0db0d4ded27f-kube-api-access-6z8p5\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw\" (UID: \"f80b76a2-b565-4c5f-b234-0db0d4ded27f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.545330 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.545213 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80b76a2-b565-4c5f-b234-0db0d4ded27f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw\" (UID: \"f80b76a2-b565-4c5f-b234-0db0d4ded27f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.545330 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.545241 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80b76a2-b565-4c5f-b234-0db0d4ded27f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw\" (UID: \"f80b76a2-b565-4c5f-b234-0db0d4ded27f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.545330 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.545280 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80b76a2-b565-4c5f-b234-0db0d4ded27f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw\" (UID: \"f80b76a2-b565-4c5f-b234-0db0d4ded27f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.545918 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.545639 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f80b76a2-b565-4c5f-b234-0db0d4ded27f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw\" (UID: \"f80b76a2-b565-4c5f-b234-0db0d4ded27f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.545918 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.545662 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f80b76a2-b565-4c5f-b234-0db0d4ded27f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw\" (UID: \"f80b76a2-b565-4c5f-b234-0db0d4ded27f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.545918 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.545688 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f80b76a2-b565-4c5f-b234-0db0d4ded27f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw\" (UID: \"f80b76a2-b565-4c5f-b234-0db0d4ded27f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.547804 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.547784 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f80b76a2-b565-4c5f-b234-0db0d4ded27f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw\" (UID: \"f80b76a2-b565-4c5f-b234-0db0d4ded27f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.548064 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.548044 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f80b76a2-b565-4c5f-b234-0db0d4ded27f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw\" (UID: \"f80b76a2-b565-4c5f-b234-0db0d4ded27f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.561767 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.561744 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z8p5\" (UniqueName: \"kubernetes.io/projected/f80b76a2-b565-4c5f-b234-0db0d4ded27f-kube-api-access-6z8p5\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw\" (UID: \"f80b76a2-b565-4c5f-b234-0db0d4ded27f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.661828 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.661754 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:21.848244 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:21.848220 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw"] Jul 06 19:16:21.848657 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:16:21.848637 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf80b76a2_b565_4c5f_b234_0db0d4ded27f.slice/crio-8af2b0aed3925fdd8a8fc4f8f5c9e53e14c836b45449be74d1a4d9409b436b62 WatchSource:0}: Error finding container 8af2b0aed3925fdd8a8fc4f8f5c9e53e14c836b45449be74d1a4d9409b436b62: Status 404 returned error can't find the container with id 8af2b0aed3925fdd8a8fc4f8f5c9e53e14c836b45449be74d1a4d9409b436b62 Jul 06 19:16:22.116717 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:22.116681 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" event={"ID":"f80b76a2-b565-4c5f-b234-0db0d4ded27f","Type":"ContainerStarted","Data":"3f26abcfd2181cee41594e1532d0824993011e4ba72c96b96690b6f991fa39a1"} Jul 06 19:16:22.116921 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:22.116901 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" event={"ID":"f80b76a2-b565-4c5f-b234-0db0d4ded27f","Type":"ContainerStarted","Data":"8af2b0aed3925fdd8a8fc4f8f5c9e53e14c836b45449be74d1a4d9409b436b62"} Jul 06 19:16:22.151797 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:22.151739 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" podStartSLOduration=0.96972182 podStartE2EDuration="1.151720499s" totalImagesPullingTime="181.998679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:22.150253476 +0000 UTC m=+1043.419435818" watchObservedRunningTime="2026-07-06 19:16:22.151720499 +0000 UTC m=+1043.420902841" Jul 06 19:16:22.703814 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:22.703785 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:22.704293 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:22.703847 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:22.721232 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:22.721193 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:23.133401 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:23.133371 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-mpm2t" Jul 06 19:16:25.023247 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:25.023214 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:16:31.662594 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:31.662556 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:31.662594 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:31.662604 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:31.674995 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:31.674972 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:16:32.177197 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:16:32.177168 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw" Jul 06 19:17:41.908782 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:17:41.908755 2553 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab81f98_e81a_4b24_beae_8babf214f1fd.slice/crio-77cab6887db8ff177b90b3c3e3a4e7234e097b6ff5c3c160ab80532b0ff992a8.scope\": RecentStats: unable to find data in memory cache]" Jul 06 19:17:42.521724 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:17:42.521688 2553 generic.go:356] "Generic (PLEG): container finished" podID="2ab81f98-e81a-4b24-beae-8babf214f1fd" containerID="77cab6887db8ff177b90b3c3e3a4e7234e097b6ff5c3c160ab80532b0ff992a8" exitCode=1 Jul 06 19:17:42.521975 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:17:42.521763 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" event={"ID":"2ab81f98-e81a-4b24-beae-8babf214f1fd","Type":"ContainerDied","Data":"77cab6887db8ff177b90b3c3e3a4e7234e097b6ff5c3c160ab80532b0ff992a8"} Jul 06 19:17:42.521975 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:17:42.521823 2553 scope.go:117] "RemoveContainer" containerID="e5cb613a78c2d82cdf9eb9680a860c2e5f059e6e5e49e631f58548371ecfd4c1" Jul 06 19:17:42.522214 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:17:42.522198 2553 scope.go:117] "RemoveContainer" containerID="77cab6887db8ff177b90b3c3e3a4e7234e097b6ff5c3c160ab80532b0ff992a8" Jul 06 19:17:42.522426 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:17:42.522392 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=authorino-operator-75cc8d5cb7-kr5j9_kuadrant-system(2ab81f98-e81a-4b24-beae-8babf214f1fd)\"" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" podUID="2ab81f98-e81a-4b24-beae-8babf214f1fd" Jul 06 19:17:45.509461 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:17:45.509429 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" Jul 06 19:17:45.509843 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:17:45.509790 2553 scope.go:117] "RemoveContainer" containerID="77cab6887db8ff177b90b3c3e3a4e7234e097b6ff5c3c160ab80532b0ff992a8" Jul 06 19:17:45.510004 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:17:45.509983 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=authorino-operator-75cc8d5cb7-kr5j9_kuadrant-system(2ab81f98-e81a-4b24-beae-8babf214f1fd)\"" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" podUID="2ab81f98-e81a-4b24-beae-8babf214f1fd" Jul 06 19:18:00.277604 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:18:00.277572 2553 scope.go:117] "RemoveContainer" containerID="77cab6887db8ff177b90b3c3e3a4e7234e097b6ff5c3c160ab80532b0ff992a8" Jul 06 19:18:00.277977 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:18:00.277759 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=authorino-operator-75cc8d5cb7-kr5j9_kuadrant-system(2ab81f98-e81a-4b24-beae-8babf214f1fd)\"" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" podUID="2ab81f98-e81a-4b24-beae-8babf214f1fd" Jul 06 19:18:01.783518 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:18:01.783488 2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" Jul 06 19:18:01.783890 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:18:01.783875 2553 scope.go:117] "RemoveContainer" containerID="77cab6887db8ff177b90b3c3e3a4e7234e097b6ff5c3c160ab80532b0ff992a8" Jul 06 19:18:01.784093 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:18:01.784073 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=authorino-operator-75cc8d5cb7-kr5j9_kuadrant-system(2ab81f98-e81a-4b24-beae-8babf214f1fd)\"" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" podUID="2ab81f98-e81a-4b24-beae-8babf214f1fd" Jul 06 19:18:16.277075 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:18:16.277044 2553 scope.go:117] "RemoveContainer" containerID="77cab6887db8ff177b90b3c3e3a4e7234e097b6ff5c3c160ab80532b0ff992a8" Jul 06 19:18:16.698388 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:18:16.698303 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" event={"ID":"2ab81f98-e81a-4b24-beae-8babf214f1fd","Type":"ContainerStarted","Data":"b92348d85c8260fb3014d83160dcc2dbaa1159f88e7462b2a06769336a4d6101"} Jul 06 19:18:16.698580 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:18:16.698562 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" Jul 06 19:18:18.758774 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:18:18.758735 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:18:27.705259 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:18:27.705175 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kr5j9" Jul 06 19:18:29.031285 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:18:29.031249 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:18:39.207924 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:18:39.207882 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:18:54.225778 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:18:54.225741 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:18:59.224228 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:18:59.224189 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:18:59.224697 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:18:59.224380 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:18:59.230375 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:18:59.230355 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:18:59.230538 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:18:59.230470 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:19:04.277050 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:19:04.277013 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:20:10.476341 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:20:10.476307 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:20:18.289019 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:20:18.288975 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:20:27.862293 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:20:27.862261 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:20:34.388301 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:20:34.388271 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:20:45.164951 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:20:45.164912 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:20:54.821686 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:20:54.821652 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:21:05.762071 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:21:05.762043 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:21:15.478246 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:21:15.478215 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:21:47.325802 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:21:47.325724 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:21:58.320201 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:21:58.320161 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:22:33.210311 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:22:33.210278 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:22:43.894906 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:22:43.894870 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:22:58.043740 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:22:58.043661 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:23:09.878459 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:23:09.878406 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:23:19.479030 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:23:19.479000 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:23:28.572577 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:23:28.572542 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:23:38.034473 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:23:38.034435 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:23:59.255830 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:23:59.255803 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:23:59.258246 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:23:59.256672 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:23:59.262210 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:23:59.262188 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:23:59.263274 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:23:59.263254 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:24:06.538946 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:24:06.538911 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s"] Jul 06 19:24:06.539372 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:24:06.539127 2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerName="manager" containerID="cri-o://001ae6fba608f47c776e68c5ddcc9bef3a2850458d00276d416bf8bb050f1cec" gracePeriod=10 Jul 06 19:24:10.453127 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:24:10.453045 2553 generic.go:356] "Generic (PLEG): container finished" podID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerID="001ae6fba608f47c776e68c5ddcc9bef3a2850458d00276d416bf8bb050f1cec" exitCode=0 Jul 06 19:24:10.453127 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:24:10.453112 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" event={"ID":"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413","Type":"ContainerDied","Data":"001ae6fba608f47c776e68c5ddcc9bef3a2850458d00276d416bf8bb050f1cec"} Jul 06 19:24:10.453586 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:24:10.453171 2553 scope.go:117] "RemoveContainer" containerID="f7daf9eca56db79d2a29a8388660184279e59db0ade798452c83a37cfa8bbd12" Jul 06 19:24:10.791583 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:24:10.791560 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:24:10.843051 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:24:10.843019 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbx78\" (UniqueName: \"kubernetes.io/projected/f889fc14-3f1f-4bf4-ba6d-3f8e5f804413-kube-api-access-kbx78\") pod \"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413\" (UID: \"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413\") " Jul 06 19:24:10.843208 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:24:10.843191 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f889fc14-3f1f-4bf4-ba6d-3f8e5f804413-extensions-socket-volume\") pod \"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413\" (UID: \"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413\") " Jul 06 19:24:10.843581 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:24:10.843558 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f889fc14-3f1f-4bf4-ba6d-3f8e5f804413-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" (UID: "f889fc14-3f1f-4bf4-ba6d-3f8e5f804413"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:24:10.845298 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:24:10.845276 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f889fc14-3f1f-4bf4-ba6d-3f8e5f804413-kube-api-access-kbx78" (OuterVolumeSpecName: "kube-api-access-kbx78") pod "f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" (UID: "f889fc14-3f1f-4bf4-ba6d-3f8e5f804413"). InnerVolumeSpecName "kube-api-access-kbx78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:24:10.944111 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:24:10.944082 2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f889fc14-3f1f-4bf4-ba6d-3f8e5f804413-extensions-socket-volume\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:24:10.944111 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:24:10.944106 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbx78\" (UniqueName: \"kubernetes.io/projected/f889fc14-3f1f-4bf4-ba6d-3f8e5f804413-kube-api-access-kbx78\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:24:11.459846 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:24:11.459811 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" event={"ID":"f889fc14-3f1f-4bf4-ba6d-3f8e5f804413","Type":"ContainerDied","Data":"2464173276dfd2980e24947f95d44570527535131f6ec96a37b292213c0ecd3b"} Jul 06 19:24:11.460245 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:24:11.459882 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s" Jul 06 19:24:11.460245 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:24:11.459886 2553 scope.go:117] "RemoveContainer" containerID="001ae6fba608f47c776e68c5ddcc9bef3a2850458d00276d416bf8bb050f1cec" Jul 06 19:24:11.487351 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:24:11.487325 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s"] Jul 06 19:24:11.495056 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:24:11.495033 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-khb2s"] Jul 06 19:24:13.282364 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:24:13.282336 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" path="/var/lib/kubelet/pods/f889fc14-3f1f-4bf4-ba6d-3f8e5f804413/volumes" Jul 06 19:25:07.718159 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.718122 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7w2nd"] Jul 06 19:25:07.718593 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.718527 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerName="manager" Jul 06 19:25:07.718593 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.718540 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerName="manager" Jul 06 19:25:07.718593 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.718550 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerName="manager" Jul 06 19:25:07.718593 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.718556 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerName="manager" Jul 06 19:25:07.718593 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.718563 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerName="manager" Jul 06 19:25:07.718593 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.718568 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerName="manager" Jul 06 19:25:07.718593 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.718581 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerName="manager" Jul 06 19:25:07.718593 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.718586 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerName="manager" Jul 06 19:25:07.718841 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.718652 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerName="manager" Jul 06 19:25:07.718841 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.718661 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerName="manager" Jul 06 19:25:07.718841 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.718666 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerName="manager" Jul 06 19:25:07.718841 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.718672 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerName="manager" Jul 06 19:25:07.721660 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.721642 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7w2nd" Jul 06 19:25:07.724692 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.724675 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xmrqf\"" Jul 06 19:25:07.741038 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.741013 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7w2nd"] Jul 06 19:25:07.834600 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.834570 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57dnn\" (UniqueName: \"kubernetes.io/projected/995c4fe8-11b1-4e0b-8190-e9c686f0a274-kube-api-access-57dnn\") pod \"kuadrant-operator-controller-manager-86878c769f-7w2nd\" (UID: \"995c4fe8-11b1-4e0b-8190-e9c686f0a274\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7w2nd" Jul 06 19:25:07.834775 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.834615 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/995c4fe8-11b1-4e0b-8190-e9c686f0a274-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7w2nd\" (UID: \"995c4fe8-11b1-4e0b-8190-e9c686f0a274\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7w2nd" Jul 06 19:25:07.935397 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.935359 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57dnn\" (UniqueName: \"kubernetes.io/projected/995c4fe8-11b1-4e0b-8190-e9c686f0a274-kube-api-access-57dnn\") pod \"kuadrant-operator-controller-manager-86878c769f-7w2nd\" (UID: \"995c4fe8-11b1-4e0b-8190-e9c686f0a274\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7w2nd" Jul 06 19:25:07.935555 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.935408 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/995c4fe8-11b1-4e0b-8190-e9c686f0a274-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7w2nd\" (UID: \"995c4fe8-11b1-4e0b-8190-e9c686f0a274\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7w2nd" Jul 06 19:25:07.935804 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.935787 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/995c4fe8-11b1-4e0b-8190-e9c686f0a274-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7w2nd\" (UID: \"995c4fe8-11b1-4e0b-8190-e9c686f0a274\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7w2nd" Jul 06 19:25:07.945602 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:07.945570 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57dnn\" (UniqueName: \"kubernetes.io/projected/995c4fe8-11b1-4e0b-8190-e9c686f0a274-kube-api-access-57dnn\") pod \"kuadrant-operator-controller-manager-86878c769f-7w2nd\" (UID: \"995c4fe8-11b1-4e0b-8190-e9c686f0a274\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7w2nd" Jul 06 19:25:08.031819 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:08.031786 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7w2nd" Jul 06 19:25:08.195944 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:08.195919 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7w2nd"] Jul 06 19:25:08.198079 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:25:08.198048 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995c4fe8_11b1_4e0b_8190_e9c686f0a274.slice/crio-a42838153276bc93efcb6830084205807adcbd01c17827ba932355a55cef2b8b WatchSource:0}: Error finding container a42838153276bc93efcb6830084205807adcbd01c17827ba932355a55cef2b8b: Status 404 returned error can't find the container with id a42838153276bc93efcb6830084205807adcbd01c17827ba932355a55cef2b8b Jul 06 19:25:08.751408 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:08.751374 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7w2nd" event={"ID":"995c4fe8-11b1-4e0b-8190-e9c686f0a274","Type":"ContainerStarted","Data":"761d8f4df162b604dc51547f9bc7fcbfa0e245cfadd80c19a530a607a9347da0"} Jul 06 19:25:08.751834 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:08.751440 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7w2nd" event={"ID":"995c4fe8-11b1-4e0b-8190-e9c686f0a274","Type":"ContainerStarted","Data":"a42838153276bc93efcb6830084205807adcbd01c17827ba932355a55cef2b8b"} Jul 06 19:25:08.751834 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:08.751592 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7w2nd" Jul 06 19:25:08.812760 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:08.812709 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7w2nd" podStartSLOduration=1.812690553 podStartE2EDuration="1.812690553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:25:08.811497646 +0000 UTC m=+1570.080679988" watchObservedRunningTime="2026-07-06 19:25:08.812690553 +0000 UTC m=+1570.081872896" Jul 06 19:25:19.759048 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:19.759017 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7w2nd" Jul 06 19:25:29.573515 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:29.573477 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:25:40.780387 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:40.780348 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:25:50.563769 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:25:50.563684 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:26:03.388909 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:26:03.388875 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:26:16.198556 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:26:16.198521 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:26:25.634943 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:26:25.634908 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:26:33.760100 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:26:33.760065 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:26:44.269352 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:26:44.269321 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:26:55.183982 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:26:55.183945 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:27:04.072549 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:27:04.072515 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:27:15.774568 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:27:15.774537 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:27:37.474649 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:27:37.474561 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:27:46.685339 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:27:46.685301 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:27:56.989165 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:27:56.989130 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:28:08.974447 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:28:08.974400 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:28:18.282874 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:28:18.282844 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:28:26.075952 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:28:26.075918 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:28:35.466431 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:28:35.466393 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:28:45.573678 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:28:45.573640 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:28:55.923226 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:28:55.923147 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:28:59.289442 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:28:59.289333 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:28:59.294819 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:28:59.290869 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:28:59.295411 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:28:59.295392 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:28:59.296764 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:28:59.296744 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:29:06.086624 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:29:06.086590 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:29:16.582296 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:29:16.582262 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:29:28.366765 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:29:28.366730 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:29:38.539288 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:29:38.539247 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:29:53.183108 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:29:53.183078 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:30:06.633072 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:30:06.633038 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:30:15.380858 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:30:15.380823 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:30:27.776961 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:30:27.776927 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:30:44.165712 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:30:44.165676 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:30:57.094081 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:30:57.094033 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:31:08.378002 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:31:08.377964 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:31:16.915672 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:31:16.915637 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:31:30.133535 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:31:30.133501 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:31:38.882735 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:31:38.882700 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:31:49.207189 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:31:49.207158 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:32:09.083313 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:32:09.083279 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:32:36.387255 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:32:36.387221 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:32:46.297842 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:32:46.297796 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:32:59.879546 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:32:59.879514 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:33:59.319674 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:33:59.319566 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:33:59.323268 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:33:59.320120 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:33:59.325949 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:33:59.325933 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:33:59.326126 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:33:59.326109 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:34:26.600659 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:34:26.600624 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:34:34.188017 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:34:34.187975 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:34:42.174954 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:34:42.174918 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:34:53.412060 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:34:53.412025 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:35:01.902434 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:35:01.902394 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:35:35.075546 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:35:35.075500 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:35:58.473312 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:35:58.473269 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:36:15.482023 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:36:15.481990 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:36:28.363062 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:36:28.362984 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:37:03.187572 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:37:03.187539 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:37:37.508034 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:37:37.507996 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:37:50.696510 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:37:50.696429 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:38:39.395113 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:38:39.395079 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:38:52.079254 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:38:52.079220 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:38:59.348539 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:38:59.348438 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:38:59.351088 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:38:59.351070 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:38:59.354383 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:38:59.354364 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:38:59.356901 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:38:59.356871 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:39:01.511473 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:39:01.511428 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:39:26.703736 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:39:26.703653 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:39:43.778966 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:39:43.778932 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:39:59.795567 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:39:59.795530 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:41:48.731254 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:41:48.731223 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:42:43.189537 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:42:43.189459 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:42:55.682713 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:42:55.682681 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:43:05.690445 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:43:05.690399 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:43:16.284998 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:43:16.284965 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:43:33.201474 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:43:33.201437 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:43:43.075296 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:43:43.075263 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:43:59.387643 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:43:59.387536 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:43:59.391299 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:43:59.388804 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:43:59.393753 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:43:59.393724 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:43:59.395172 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:43:59.395155 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:44:10.375987 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:10.375954 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:44:43.779353 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:43.779316 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj"] Jul 06 19:44:43.779831 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:43.779709 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerName="manager" Jul 06 19:44:43.779831 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:43.779721 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerName="manager" Jul 06 19:44:43.779831 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:43.779792 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f889fc14-3f1f-4bf4-ba6d-3f8e5f804413" containerName="manager" Jul 06 19:44:43.782864 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:43.782848 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:43.786011 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:43.785989 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5f29b8\"/\"openshift-service-ca.crt\"" Jul 06 19:44:43.786296 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:43.786274 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5f29b8\"/\"auth-test-model-5f29b8-kserve-self-signed-certs\"" Jul 06 19:44:43.786381 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:43.786274 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5f29b8\"/\"kube-root-ca.crt\"" Jul 06 19:44:43.793729 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:43.793707 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj"] Jul 06 19:44:43.907199 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:43.907156 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc9ks\" (UniqueName: \"kubernetes.io/projected/cd646fbb-a3f8-483c-9004-dba113967de4-kube-api-access-pc9ks\") pod \"auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:43.907199 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:43.907200 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-home\") pod \"auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:43.907487 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:43.907245 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-model-cache\") pod \"auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:43.907487 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:43.907331 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-dshm\") pod \"auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:43.907487 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:43.907352 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd646fbb-a3f8-483c-9004-dba113967de4-tls-certs\") pod \"auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:43.907669 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:43.907491 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-tmp-dir\") pod \"auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:44.008367 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:44.008328 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-dshm\") pod \"auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:44.008367 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:44.008364 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd646fbb-a3f8-483c-9004-dba113967de4-tls-certs\") pod \"auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:44.008666 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:44.008402 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-tmp-dir\") pod \"auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:44.008666 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:44.008471 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc9ks\" (UniqueName: \"kubernetes.io/projected/cd646fbb-a3f8-483c-9004-dba113967de4-kube-api-access-pc9ks\") pod \"auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:44.008851 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:44.008825 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-home\") pod \"auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:44.008941 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:44.008858 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-tmp-dir\") pod \"auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:44.008941 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:44.008902 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-model-cache\") pod \"auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:44.009207 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:44.009175 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-home\") pod \"auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:44.009207 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:44.009201 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-model-cache\") pod \"auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:44.011545 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:44.011513 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-dshm\") pod \"auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:44.011676 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:44.011661 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd646fbb-a3f8-483c-9004-dba113967de4-tls-certs\") pod \"auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:44.017075 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:44.017050 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc9ks\" (UniqueName: \"kubernetes.io/projected/cd646fbb-a3f8-483c-9004-dba113967de4-kube-api-access-pc9ks\") pod \"auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:44.093275 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:44.093221 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:44.230818 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:44.230793 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj"] Jul 06 19:44:44.232784 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:44:44.232758 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd646fbb_a3f8_483c_9004_dba113967de4.slice/crio-6246a5ed2bcf999a8a6b00e1676b5b0e26163a0b5df9df57cb623308002e00c9 WatchSource:0}: Error finding container 6246a5ed2bcf999a8a6b00e1676b5b0e26163a0b5df9df57cb623308002e00c9: Status 404 returned error can't find the container with id 6246a5ed2bcf999a8a6b00e1676b5b0e26163a0b5df9df57cb623308002e00c9 Jul 06 19:44:44.563983 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:44.563948 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" event={"ID":"cd646fbb-a3f8-483c-9004-dba113967de4","Type":"ContainerStarted","Data":"dcba2719e06abb103e17ac6ee09f7dc8f5fe7c3c077d5648eea9cab94f56375b"} Jul 06 19:44:44.563983 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:44.563992 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" event={"ID":"cd646fbb-a3f8-483c-9004-dba113967de4","Type":"ContainerStarted","Data":"6246a5ed2bcf999a8a6b00e1676b5b0e26163a0b5df9df57cb623308002e00c9"} Jul 06 19:44:44.588619 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:44.588575 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" podStartSLOduration=1.5885607940000002 podStartE2EDuration="1.588560794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:44:44.587742269 +0000 UTC m=+2745.856924614" watchObservedRunningTime="2026-07-06 19:44:44.588560794 +0000 UTC m=+2745.857743135" Jul 06 19:44:47.662733 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:47.662667 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:44:54.094246 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:54.094211 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:54.094659 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:54.094295 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:54.107530 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:54.107509 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:44:54.626905 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:44:54.626878 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:45:00.141067 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:00.141031 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-5c92ad-29722785-zn4xm"] Jul 06 19:45:00.144877 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:00.144852 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-5c92ad-29722785-zn4xm" Jul 06 19:45:00.147657 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:00.147639 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-zxgdj\"" Jul 06 19:45:00.160569 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:00.160544 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-5c92ad-29722785-zn4xm"] Jul 06 19:45:00.258312 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:00.258281 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqqhq\" (UniqueName: \"kubernetes.io/projected/724e5c31-c548-449c-a376-43916cc5c0a2-kube-api-access-sqqhq\") pod \"maas-api-key-cleanup-e2e-shared-a-5c92ad-29722785-zn4xm\" (UID: \"724e5c31-c548-449c-a376-43916cc5c0a2\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-5c92ad-29722785-zn4xm" Jul 06 19:45:00.359875 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:00.359837 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqqhq\" (UniqueName: \"kubernetes.io/projected/724e5c31-c548-449c-a376-43916cc5c0a2-kube-api-access-sqqhq\") pod \"maas-api-key-cleanup-e2e-shared-a-5c92ad-29722785-zn4xm\" (UID: \"724e5c31-c548-449c-a376-43916cc5c0a2\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-5c92ad-29722785-zn4xm" Jul 06 19:45:00.368577 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:00.368552 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqqhq\" (UniqueName: \"kubernetes.io/projected/724e5c31-c548-449c-a376-43916cc5c0a2-kube-api-access-sqqhq\") pod \"maas-api-key-cleanup-e2e-shared-a-5c92ad-29722785-zn4xm\" (UID: \"724e5c31-c548-449c-a376-43916cc5c0a2\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-5c92ad-29722785-zn4xm" Jul 06 19:45:00.454410 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:00.454335 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-5c92ad-29722785-zn4xm" Jul 06 19:45:00.595986 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:00.595962 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-5c92ad-29722785-zn4xm"] Jul 06 19:45:00.598304 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:45:00.598267 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724e5c31_c548_449c_a376_43916cc5c0a2.slice/crio-1b5e6e09988db0e7c10a37b0ba9a80d9a70fdbec493eca8b5dfa9a0dba45f1cd WatchSource:0}: Error finding container 1b5e6e09988db0e7c10a37b0ba9a80d9a70fdbec493eca8b5dfa9a0dba45f1cd: Status 404 returned error can't find the container with id 1b5e6e09988db0e7c10a37b0ba9a80d9a70fdbec493eca8b5dfa9a0dba45f1cd Jul 06 19:45:00.647334 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:00.647310 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-5c92ad-29722785-zn4xm" event={"ID":"724e5c31-c548-449c-a376-43916cc5c0a2","Type":"ContainerStarted","Data":"1b5e6e09988db0e7c10a37b0ba9a80d9a70fdbec493eca8b5dfa9a0dba45f1cd"} Jul 06 19:45:01.653987 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:01.653956 2553 generic.go:356] "Generic (PLEG): container finished" podID="724e5c31-c548-449c-a376-43916cc5c0a2" containerID="65737fefcf3e1d0cecb5c03978be00c47999da311ca02ab0002a059f4827bf15" exitCode=0 Jul 06 19:45:01.654380 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:01.654053 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-5c92ad-29722785-zn4xm" event={"ID":"724e5c31-c548-449c-a376-43916cc5c0a2","Type":"ContainerDied","Data":"65737fefcf3e1d0cecb5c03978be00c47999da311ca02ab0002a059f4827bf15"} Jul 06 19:45:02.170528 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:02.170496 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:45:02.798772 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:02.798744 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-5c92ad-29722785-zn4xm" Jul 06 19:45:02.883613 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:02.883584 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqqhq\" (UniqueName: \"kubernetes.io/projected/724e5c31-c548-449c-a376-43916cc5c0a2-kube-api-access-sqqhq\") pod \"724e5c31-c548-449c-a376-43916cc5c0a2\" (UID: \"724e5c31-c548-449c-a376-43916cc5c0a2\") " Jul 06 19:45:02.885777 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:02.885754 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724e5c31-c548-449c-a376-43916cc5c0a2-kube-api-access-sqqhq" (OuterVolumeSpecName: "kube-api-access-sqqhq") pod "724e5c31-c548-449c-a376-43916cc5c0a2" (UID: "724e5c31-c548-449c-a376-43916cc5c0a2"). InnerVolumeSpecName "kube-api-access-sqqhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:45:02.985026 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:02.984991 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqqhq\" (UniqueName: \"kubernetes.io/projected/724e5c31-c548-449c-a376-43916cc5c0a2-kube-api-access-sqqhq\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:45:03.664945 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:03.664870 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-5c92ad-29722785-zn4xm" Jul 06 19:45:03.664945 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:03.664897 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-5c92ad-29722785-zn4xm" event={"ID":"724e5c31-c548-449c-a376-43916cc5c0a2","Type":"ContainerDied","Data":"1b5e6e09988db0e7c10a37b0ba9a80d9a70fdbec493eca8b5dfa9a0dba45f1cd"} Jul 06 19:45:03.664945 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:03.664937 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b5e6e09988db0e7c10a37b0ba9a80d9a70fdbec493eca8b5dfa9a0dba45f1cd" Jul 06 19:45:20.872032 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:20.871951 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:45:39.306405 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:39.306374 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:45:57.482568 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:45:57.482536 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:46:13.376905 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:46:13.376871 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:46:31.966902 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:46:31.966868 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:46:47.175043 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:46:47.175010 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj"] Jul 06 19:46:50.969205 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:46:50.969126 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:47:02.195322 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.195247 2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" podUID="cd646fbb-a3f8-483c-9004-dba113967de4" containerName="main" containerID="cri-o://dcba2719e06abb103e17ac6ee09f7dc8f5fe7c3c077d5648eea9cab94f56375b" gracePeriod=45 Jul 06 19:47:02.451866 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.451814 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:47:02.555793 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.555756 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pc9ks\" (UniqueName: \"kubernetes.io/projected/cd646fbb-a3f8-483c-9004-dba113967de4-kube-api-access-pc9ks\") pod \"cd646fbb-a3f8-483c-9004-dba113967de4\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " Jul 06 19:47:02.555793 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.555793 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-model-cache\") pod \"cd646fbb-a3f8-483c-9004-dba113967de4\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " Jul 06 19:47:02.556043 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.555821 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-tmp-dir\") pod \"cd646fbb-a3f8-483c-9004-dba113967de4\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " Jul 06 19:47:02.556043 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.555892 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd646fbb-a3f8-483c-9004-dba113967de4-tls-certs\") pod \"cd646fbb-a3f8-483c-9004-dba113967de4\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " Jul 06 19:47:02.556043 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.555951 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-home\") pod \"cd646fbb-a3f8-483c-9004-dba113967de4\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " Jul 06 19:47:02.556043 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.555976 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-dshm\") pod \"cd646fbb-a3f8-483c-9004-dba113967de4\" (UID: \"cd646fbb-a3f8-483c-9004-dba113967de4\") " Jul 06 19:47:02.556255 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.556091 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd646fbb-a3f8-483c-9004-dba113967de4" (UID: "cd646fbb-a3f8-483c-9004-dba113967de4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:47:02.556255 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.556130 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd646fbb-a3f8-483c-9004-dba113967de4" (UID: "cd646fbb-a3f8-483c-9004-dba113967de4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:47:02.556255 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.556230 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-home" (OuterVolumeSpecName: "home") pod "cd646fbb-a3f8-483c-9004-dba113967de4" (UID: "cd646fbb-a3f8-483c-9004-dba113967de4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:47:02.556406 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.556342 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-home\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:47:02.556406 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.556363 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-model-cache\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:47:02.556406 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.556395 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-tmp-dir\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:47:02.558157 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.558131 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd646fbb-a3f8-483c-9004-dba113967de4-kube-api-access-pc9ks" (OuterVolumeSpecName: "kube-api-access-pc9ks") pod "cd646fbb-a3f8-483c-9004-dba113967de4" (UID: "cd646fbb-a3f8-483c-9004-dba113967de4"). InnerVolumeSpecName "kube-api-access-pc9ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:47:02.558258 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.558172 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd646fbb-a3f8-483c-9004-dba113967de4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd646fbb-a3f8-483c-9004-dba113967de4" (UID: "cd646fbb-a3f8-483c-9004-dba113967de4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:47:02.558258 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.558179 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-dshm" (OuterVolumeSpecName: "dshm") pod "cd646fbb-a3f8-483c-9004-dba113967de4" (UID: "cd646fbb-a3f8-483c-9004-dba113967de4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:47:02.657463 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.657433 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd646fbb-a3f8-483c-9004-dba113967de4-tls-certs\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:47:02.657463 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.657457 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd646fbb-a3f8-483c-9004-dba113967de4-dshm\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:47:02.657463 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:02.657465 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc9ks\" (UniqueName: \"kubernetes.io/projected/cd646fbb-a3f8-483c-9004-dba113967de4-kube-api-access-pc9ks\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:47:03.276144 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:03.276108 2553 generic.go:356] "Generic (PLEG): container finished" podID="cd646fbb-a3f8-483c-9004-dba113967de4" containerID="dcba2719e06abb103e17ac6ee09f7dc8f5fe7c3c077d5648eea9cab94f56375b" exitCode=0 Jul 06 19:47:03.276724 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:03.276153 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" Jul 06 19:47:03.276724 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:03.276196 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" event={"ID":"cd646fbb-a3f8-483c-9004-dba113967de4","Type":"ContainerDied","Data":"dcba2719e06abb103e17ac6ee09f7dc8f5fe7c3c077d5648eea9cab94f56375b"} Jul 06 19:47:03.276724 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:03.276241 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj" event={"ID":"cd646fbb-a3f8-483c-9004-dba113967de4","Type":"ContainerDied","Data":"6246a5ed2bcf999a8a6b00e1676b5b0e26163a0b5df9df57cb623308002e00c9"} Jul 06 19:47:03.276724 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:03.276309 2553 scope.go:117] "RemoveContainer" containerID="dcba2719e06abb103e17ac6ee09f7dc8f5fe7c3c077d5648eea9cab94f56375b" Jul 06 19:47:03.286217 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:03.286197 2553 scope.go:117] "RemoveContainer" containerID="dcba2719e06abb103e17ac6ee09f7dc8f5fe7c3c077d5648eea9cab94f56375b" Jul 06 19:47:03.286487 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:47:03.286466 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcba2719e06abb103e17ac6ee09f7dc8f5fe7c3c077d5648eea9cab94f56375b\": container with ID starting with dcba2719e06abb103e17ac6ee09f7dc8f5fe7c3c077d5648eea9cab94f56375b not found: ID does not exist" containerID="dcba2719e06abb103e17ac6ee09f7dc8f5fe7c3c077d5648eea9cab94f56375b" Jul 06 19:47:03.286597 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:03.286493 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcba2719e06abb103e17ac6ee09f7dc8f5fe7c3c077d5648eea9cab94f56375b"} err="failed to get container status \"dcba2719e06abb103e17ac6ee09f7dc8f5fe7c3c077d5648eea9cab94f56375b\": rpc error: code = NotFound desc = could not find container \"dcba2719e06abb103e17ac6ee09f7dc8f5fe7c3c077d5648eea9cab94f56375b\": container with ID starting with dcba2719e06abb103e17ac6ee09f7dc8f5fe7c3c077d5648eea9cab94f56375b not found: ID does not exist" Jul 06 19:47:03.300276 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:03.300248 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj"] Jul 06 19:47:03.305541 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:03.305517 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5f29b8/auth-test-model-5f29b8-kserve-5c4c9b8bd7-8nqlj"] Jul 06 19:47:05.283244 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:05.283204 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd646fbb-a3f8-483c-9004-dba113967de4" path="/var/lib/kubelet/pods/cd646fbb-a3f8-483c-9004-dba113967de4/volumes" Jul 06 19:47:16.599916 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:16.599879 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:47:33.985903 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:33.985867 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5"] Jul 06 19:47:33.986440 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:33.986408 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd646fbb-a3f8-483c-9004-dba113967de4" containerName="main" Jul 06 19:47:33.986440 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:33.986442 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd646fbb-a3f8-483c-9004-dba113967de4" containerName="main" Jul 06 19:47:33.986535 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:33.986453 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="724e5c31-c548-449c-a376-43916cc5c0a2" containerName="cleanup" Jul 06 19:47:33.986535 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:33.986462 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="724e5c31-c548-449c-a376-43916cc5c0a2" containerName="cleanup" Jul 06 19:47:33.986602 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:33.986563 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd646fbb-a3f8-483c-9004-dba113967de4" containerName="main" Jul 06 19:47:33.986602 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:33.986573 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="724e5c31-c548-449c-a376-43916cc5c0a2" containerName="cleanup" Jul 06 19:47:33.989766 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:33.989743 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:33.992296 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:33.992274 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5f29b8\"/\"kube-root-ca.crt\"" Jul 06 19:47:33.992442 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:33.992371 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5f29b8\"/\"openshift-service-ca.crt\"" Jul 06 19:47:33.992912 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:33.992892 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5f29b8\"/\"default-dockercfg-j9f99\"" Jul 06 19:47:33.993030 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:33.993010 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5f29b8\"/\"sub-test-model-5f29b8-kserve-self-signed-certs\"" Jul 06 19:47:34.002613 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.002588 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5"] Jul 06 19:47:34.147274 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.147233 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-dshm\") pod \"sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:34.147274 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.147277 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-model-cache\") pod \"sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:34.147530 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.147302 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztrkg\" (UniqueName: \"kubernetes.io/projected/0d3aa21d-0163-4083-accd-5a94a1597c31-kube-api-access-ztrkg\") pod \"sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:34.147530 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.147374 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3aa21d-0163-4083-accd-5a94a1597c31-tls-certs\") pod \"sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:34.147530 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.147459 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-home\") pod \"sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:34.147530 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.147477 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-tmp-dir\") pod \"sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:34.248339 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.248241 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-dshm\") pod \"sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:34.248339 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.248298 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-model-cache\") pod \"sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:34.248339 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.248324 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztrkg\" (UniqueName: \"kubernetes.io/projected/0d3aa21d-0163-4083-accd-5a94a1597c31-kube-api-access-ztrkg\") pod \"sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:34.248667 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.248363 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3aa21d-0163-4083-accd-5a94a1597c31-tls-certs\") pod \"sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:34.248667 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.248452 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-home\") pod \"sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:34.248667 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.248475 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-tmp-dir\") pod \"sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:34.248827 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.248804 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-model-cache\") pod \"sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:34.248827 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.248819 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-home\") pod \"sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:34.248903 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.248847 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-tmp-dir\") pod \"sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:34.250707 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.250687 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-dshm\") pod \"sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:34.250989 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.250972 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3aa21d-0163-4083-accd-5a94a1597c31-tls-certs\") pod \"sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:34.258494 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.258475 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztrkg\" (UniqueName: \"kubernetes.io/projected/0d3aa21d-0163-4083-accd-5a94a1597c31-kube-api-access-ztrkg\") pod \"sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:34.299253 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.299225 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:34.451155 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:34.451101 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5"] Jul 06 19:47:34.454011 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:47:34.453985 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d3aa21d_0163_4083_accd_5a94a1597c31.slice/crio-e30b28e6cf2f03e725a7c09d4635add27632eb00cb2dd8dfa844356041192889 WatchSource:0}: Error finding container e30b28e6cf2f03e725a7c09d4635add27632eb00cb2dd8dfa844356041192889: Status 404 returned error can't find the container with id e30b28e6cf2f03e725a7c09d4635add27632eb00cb2dd8dfa844356041192889 Jul 06 19:47:35.457977 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:35.457935 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" event={"ID":"0d3aa21d-0163-4083-accd-5a94a1597c31","Type":"ContainerStarted","Data":"c2259a2b12c340c4c8b6e0a95a884b2e51fc81b5ac6c81d1feb436e3efe8e435"} Jul 06 19:47:35.458368 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:35.457986 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" event={"ID":"0d3aa21d-0163-4083-accd-5a94a1597c31","Type":"ContainerStarted","Data":"e30b28e6cf2f03e725a7c09d4635add27632eb00cb2dd8dfa844356041192889"} Jul 06 19:47:35.482199 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:35.482151 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" podStartSLOduration=2.482135805 podStartE2EDuration="2.482135805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:47:35.480769201 +0000 UTC m=+2916.749951555" watchObservedRunningTime="2026-07-06 19:47:35.482135805 +0000 UTC m=+2916.751318147" Jul 06 19:47:38.400825 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:38.400792 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:47:44.300132 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:44.300090 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:44.300632 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:44.300143 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:44.313137 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:44.313116 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:47:44.512956 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:47:44.512922 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:48:06.880518 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:06.880480 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:48:23.856237 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:23.856163 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5"] Jul 06 19:48:27.982102 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:27.982062 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:48:38.876715 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:38.876678 2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" podUID="0d3aa21d-0163-4083-accd-5a94a1597c31" containerName="main" containerID="cri-o://c2259a2b12c340c4c8b6e0a95a884b2e51fc81b5ac6c81d1feb436e3efe8e435" gracePeriod=45 Jul 06 19:48:39.126276 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.126251 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:48:39.219750 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.219726 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3aa21d-0163-4083-accd-5a94a1597c31-tls-certs\") pod \"0d3aa21d-0163-4083-accd-5a94a1597c31\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " Jul 06 19:48:39.219913 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.219773 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztrkg\" (UniqueName: \"kubernetes.io/projected/0d3aa21d-0163-4083-accd-5a94a1597c31-kube-api-access-ztrkg\") pod \"0d3aa21d-0163-4083-accd-5a94a1597c31\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " Jul 06 19:48:39.219913 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.219811 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-home\") pod \"0d3aa21d-0163-4083-accd-5a94a1597c31\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " Jul 06 19:48:39.219913 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.219856 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-dshm\") pod \"0d3aa21d-0163-4083-accd-5a94a1597c31\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " Jul 06 19:48:39.219913 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.219870 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-model-cache\") pod \"0d3aa21d-0163-4083-accd-5a94a1597c31\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " Jul 06 19:48:39.220117 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.219927 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-tmp-dir\") pod \"0d3aa21d-0163-4083-accd-5a94a1597c31\" (UID: \"0d3aa21d-0163-4083-accd-5a94a1597c31\") " Jul 06 19:48:39.220209 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.220168 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-home" (OuterVolumeSpecName: "home") pod "0d3aa21d-0163-4083-accd-5a94a1597c31" (UID: "0d3aa21d-0163-4083-accd-5a94a1597c31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:48:39.220209 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.220177 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d3aa21d-0163-4083-accd-5a94a1597c31" (UID: "0d3aa21d-0163-4083-accd-5a94a1597c31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:48:39.220319 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.220204 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d3aa21d-0163-4083-accd-5a94a1597c31" (UID: "0d3aa21d-0163-4083-accd-5a94a1597c31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:48:39.220386 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.220368 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-tmp-dir\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:48:39.220460 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.220394 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-home\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:48:39.220460 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.220409 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-model-cache\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:48:39.222080 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.222060 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-dshm" (OuterVolumeSpecName: "dshm") pod "0d3aa21d-0163-4083-accd-5a94a1597c31" (UID: "0d3aa21d-0163-4083-accd-5a94a1597c31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:48:39.222173 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.222103 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d3aa21d-0163-4083-accd-5a94a1597c31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d3aa21d-0163-4083-accd-5a94a1597c31" (UID: "0d3aa21d-0163-4083-accd-5a94a1597c31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:48:39.222173 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.222148 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3aa21d-0163-4083-accd-5a94a1597c31-kube-api-access-ztrkg" (OuterVolumeSpecName: "kube-api-access-ztrkg") pod "0d3aa21d-0163-4083-accd-5a94a1597c31" (UID: "0d3aa21d-0163-4083-accd-5a94a1597c31"). InnerVolumeSpecName "kube-api-access-ztrkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:48:39.321465 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.321436 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3aa21d-0163-4083-accd-5a94a1597c31-dshm\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:48:39.321465 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.321461 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3aa21d-0163-4083-accd-5a94a1597c31-tls-certs\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:48:39.321638 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.321471 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztrkg\" (UniqueName: \"kubernetes.io/projected/0d3aa21d-0163-4083-accd-5a94a1597c31-kube-api-access-ztrkg\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:48:39.787613 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.787577 2553 generic.go:356] "Generic (PLEG): container finished" podID="0d3aa21d-0163-4083-accd-5a94a1597c31" containerID="c2259a2b12c340c4c8b6e0a95a884b2e51fc81b5ac6c81d1feb436e3efe8e435" exitCode=0 Jul 06 19:48:39.787811 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.787622 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" Jul 06 19:48:39.787811 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.787638 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" event={"ID":"0d3aa21d-0163-4083-accd-5a94a1597c31","Type":"ContainerDied","Data":"c2259a2b12c340c4c8b6e0a95a884b2e51fc81b5ac6c81d1feb436e3efe8e435"} Jul 06 19:48:39.787811 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.787680 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5" event={"ID":"0d3aa21d-0163-4083-accd-5a94a1597c31","Type":"ContainerDied","Data":"e30b28e6cf2f03e725a7c09d4635add27632eb00cb2dd8dfa844356041192889"} Jul 06 19:48:39.787811 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.787696 2553 scope.go:117] "RemoveContainer" containerID="c2259a2b12c340c4c8b6e0a95a884b2e51fc81b5ac6c81d1feb436e3efe8e435" Jul 06 19:48:39.796503 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.796485 2553 scope.go:117] "RemoveContainer" containerID="c2259a2b12c340c4c8b6e0a95a884b2e51fc81b5ac6c81d1feb436e3efe8e435" Jul 06 19:48:39.796733 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:48:39.796715 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2259a2b12c340c4c8b6e0a95a884b2e51fc81b5ac6c81d1feb436e3efe8e435\": container with ID starting with c2259a2b12c340c4c8b6e0a95a884b2e51fc81b5ac6c81d1feb436e3efe8e435 not found: ID does not exist" containerID="c2259a2b12c340c4c8b6e0a95a884b2e51fc81b5ac6c81d1feb436e3efe8e435" Jul 06 19:48:39.796806 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.796746 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2259a2b12c340c4c8b6e0a95a884b2e51fc81b5ac6c81d1feb436e3efe8e435"} err="failed to get container status \"c2259a2b12c340c4c8b6e0a95a884b2e51fc81b5ac6c81d1feb436e3efe8e435\": rpc error: code = NotFound desc = could not find container \"c2259a2b12c340c4c8b6e0a95a884b2e51fc81b5ac6c81d1feb436e3efe8e435\": container with ID starting with c2259a2b12c340c4c8b6e0a95a884b2e51fc81b5ac6c81d1feb436e3efe8e435 not found: ID does not exist" Jul 06 19:48:39.802857 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.802835 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5"] Jul 06 19:48:39.809838 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:39.809813 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5f29b8/sub-test-model-5f29b8-kserve-5c4c478f6c-svwb5"] Jul 06 19:48:41.282377 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:41.282343 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d3aa21d-0163-4083-accd-5a94a1597c31" path="/var/lib/kubelet/pods/0d3aa21d-0163-4083-accd-5a94a1597c31/volumes" Jul 06 19:48:46.545814 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:46.545779 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:48:59.419037 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:59.418942 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:48:59.422227 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:59.422209 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:48:59.424636 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:59.424617 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:48:59.427995 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:48:59.427981 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x5g5z_d422e92a-be96-4d4a-8011-66ff2945286d/ovn-acl-logging/0.log" Jul 06 19:49:32.377777 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:49:32.377734 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:49:46.990144 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:49:46.990102 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:50:07.043935 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:07.043896 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:50:12.143593 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:12.143560 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7551-140020560225312"] Jul 06 19:50:12.144118 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:12.144097 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d3aa21d-0163-4083-accd-5a94a1597c31" containerName="main" Jul 06 19:50:12.144199 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:12.144122 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d3aa21d-0163-4083-accd-5a94a1597c31" containerName="main" Jul 06 19:50:12.144254 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:12.144228 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d3aa21d-0163-4083-accd-5a94a1597c31" containerName="main" Jul 06 19:50:12.147191 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:12.147170 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560225312" Jul 06 19:50:12.149455 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:12.149411 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jlb4q\"" Jul 06 19:50:12.153105 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:12.153082 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7551-140020560225312"] Jul 06 19:50:12.238750 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:12.238714 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzhdz\" (UniqueName: \"kubernetes.io/projected/41d81419-2f75-4113-ab2f-6c20320ece0d-kube-api-access-pzhdz\") pod \"test-curl-7551-140020560225312\" (UID: \"41d81419-2f75-4113-ab2f-6c20320ece0d\") " pod="opendatahub/test-curl-7551-140020560225312" Jul 06 19:50:12.339835 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:12.339796 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzhdz\" (UniqueName: \"kubernetes.io/projected/41d81419-2f75-4113-ab2f-6c20320ece0d-kube-api-access-pzhdz\") pod \"test-curl-7551-140020560225312\" (UID: \"41d81419-2f75-4113-ab2f-6c20320ece0d\") " pod="opendatahub/test-curl-7551-140020560225312" Jul 06 19:50:12.352210 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:12.352174 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzhdz\" (UniqueName: \"kubernetes.io/projected/41d81419-2f75-4113-ab2f-6c20320ece0d-kube-api-access-pzhdz\") pod \"test-curl-7551-140020560225312\" (UID: \"41d81419-2f75-4113-ab2f-6c20320ece0d\") " pod="opendatahub/test-curl-7551-140020560225312" Jul 06 19:50:12.457403 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:12.457322 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560225312" Jul 06 19:50:12.595625 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:12.595594 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7551-140020560225312"] Jul 06 19:50:12.598553 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:50:12.598527 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41d81419_2f75_4113_ab2f_6c20320ece0d.slice/crio-e4e8b5f9e990c891cd285580721db42cf043f03a8894bdb66abe0b5302ff3c42 WatchSource:0}: Error finding container e4e8b5f9e990c891cd285580721db42cf043f03a8894bdb66abe0b5302ff3c42: Status 404 returned error can't find the container with id e4e8b5f9e990c891cd285580721db42cf043f03a8894bdb66abe0b5302ff3c42 Jul 06 19:50:12.599981 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:12.599962 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:50:13.271562 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:13.271520 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7551-140020560225312" event={"ID":"41d81419-2f75-4113-ab2f-6c20320ece0d","Type":"ContainerStarted","Data":"e4e8b5f9e990c891cd285580721db42cf043f03a8894bdb66abe0b5302ff3c42"} Jul 06 19:50:14.277624 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:14.277594 2553 generic.go:356] "Generic (PLEG): container finished" podID="41d81419-2f75-4113-ab2f-6c20320ece0d" containerID="5dde789e469939edef48bbac57ffd2881c2906023705299b7af4c1165df510b2" exitCode=0 Jul 06 19:50:14.278002 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:14.277656 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7551-140020560225312" event={"ID":"41d81419-2f75-4113-ab2f-6c20320ece0d","Type":"ContainerDied","Data":"5dde789e469939edef48bbac57ffd2881c2906023705299b7af4c1165df510b2"} Jul 06 19:50:15.423055 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:15.423035 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560225312" Jul 06 19:50:15.461233 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:15.461211 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7551-140020560225312_41d81419-2f75-4113-ab2f-6c20320ece0d/test-curl-7551-140020560225312/0.log" Jul 06 19:50:15.469411 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:15.469386 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzhdz\" (UniqueName: \"kubernetes.io/projected/41d81419-2f75-4113-ab2f-6c20320ece0d-kube-api-access-pzhdz\") pod \"41d81419-2f75-4113-ab2f-6c20320ece0d\" (UID: \"41d81419-2f75-4113-ab2f-6c20320ece0d\") " Jul 06 19:50:15.471605 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:15.471583 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d81419-2f75-4113-ab2f-6c20320ece0d-kube-api-access-pzhdz" (OuterVolumeSpecName: "kube-api-access-pzhdz") pod "41d81419-2f75-4113-ab2f-6c20320ece0d" (UID: "41d81419-2f75-4113-ab2f-6c20320ece0d"). InnerVolumeSpecName "kube-api-access-pzhdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:50:15.484804 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:15.484783 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7551-140020560225312"] Jul 06 19:50:15.488581 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:15.488560 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7551-140020560225312"] Jul 06 19:50:15.571155 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:15.571085 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzhdz\" (UniqueName: \"kubernetes.io/projected/41d81419-2f75-4113-ab2f-6c20320ece0d-kube-api-access-pzhdz\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:50:15.625055 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:15.625027 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7551-140020560225424"] Jul 06 19:50:15.625441 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:15.625427 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d81419-2f75-4113-ab2f-6c20320ece0d" containerName="test-curl-7551-140020560225312" Jul 06 19:50:15.625490 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:15.625443 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d81419-2f75-4113-ab2f-6c20320ece0d" containerName="test-curl-7551-140020560225312" Jul 06 19:50:15.625526 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:15.625520 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d81419-2f75-4113-ab2f-6c20320ece0d" containerName="test-curl-7551-140020560225312" Jul 06 19:50:15.628460 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:15.628442 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560225424" Jul 06 19:50:15.635392 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:15.635360 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7551-140020560225424"] Jul 06 19:50:15.671533 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:15.671507 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g89ss\" (UniqueName: \"kubernetes.io/projected/8a8ff464-0dd2-4a74-95ee-4813783ba1ef-kube-api-access-g89ss\") pod \"test-curl-7551-140020560225424\" (UID: \"8a8ff464-0dd2-4a74-95ee-4813783ba1ef\") " pod="opendatahub/test-curl-7551-140020560225424" Jul 06 19:50:15.773060 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:15.773034 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g89ss\" (UniqueName: \"kubernetes.io/projected/8a8ff464-0dd2-4a74-95ee-4813783ba1ef-kube-api-access-g89ss\") pod \"test-curl-7551-140020560225424\" (UID: \"8a8ff464-0dd2-4a74-95ee-4813783ba1ef\") " pod="opendatahub/test-curl-7551-140020560225424" Jul 06 19:50:15.781023 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:15.780997 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g89ss\" (UniqueName: \"kubernetes.io/projected/8a8ff464-0dd2-4a74-95ee-4813783ba1ef-kube-api-access-g89ss\") pod \"test-curl-7551-140020560225424\" (UID: \"8a8ff464-0dd2-4a74-95ee-4813783ba1ef\") " pod="opendatahub/test-curl-7551-140020560225424" Jul 06 19:50:15.937686 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:15.937601 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560225424" Jul 06 19:50:16.068690 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:16.068664 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7551-140020560225424"] Jul 06 19:50:16.071291 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:50:16.071266 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a8ff464_0dd2_4a74_95ee_4813783ba1ef.slice/crio-ab1a57ea7f801a5be3c4ca637962edab9ab12f0f56063256453e27d0d339ed32 WatchSource:0}: Error finding container ab1a57ea7f801a5be3c4ca637962edab9ab12f0f56063256453e27d0d339ed32: Status 404 returned error can't find the container with id ab1a57ea7f801a5be3c4ca637962edab9ab12f0f56063256453e27d0d339ed32 Jul 06 19:50:16.288683 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:16.288659 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560225312" Jul 06 19:50:16.288856 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:16.288665 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4e8b5f9e990c891cd285580721db42cf043f03a8894bdb66abe0b5302ff3c42" Jul 06 19:50:16.289818 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:16.289792 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7551-140020560225424" event={"ID":"8a8ff464-0dd2-4a74-95ee-4813783ba1ef","Type":"ContainerStarted","Data":"ab1a57ea7f801a5be3c4ca637962edab9ab12f0f56063256453e27d0d339ed32"} Jul 06 19:50:16.298984 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:50:16.298959 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7551-140020560225312\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="41d81419-2f75-4113-ab2f-6c20320ece0d" pod="opendatahub/test-curl-7551-140020560225312" Jul 06 19:50:17.283377 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:17.283345 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41d81419-2f75-4113-ab2f-6c20320ece0d" path="/var/lib/kubelet/pods/41d81419-2f75-4113-ab2f-6c20320ece0d/volumes" Jul 06 19:50:17.296252 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:17.296223 2553 generic.go:356] "Generic (PLEG): container finished" podID="8a8ff464-0dd2-4a74-95ee-4813783ba1ef" containerID="2e6e7a8737f28ab00e12f0f0088d848df7124baa872b2b0553830562552cc058" exitCode=0 Jul 06 19:50:17.296425 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:17.296315 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7551-140020560225424" event={"ID":"8a8ff464-0dd2-4a74-95ee-4813783ba1ef","Type":"ContainerDied","Data":"2e6e7a8737f28ab00e12f0f0088d848df7124baa872b2b0553830562552cc058"} Jul 06 19:50:18.443517 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:18.443495 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560225424" Jul 06 19:50:18.474670 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:18.474650 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7551-140020560225424_8a8ff464-0dd2-4a74-95ee-4813783ba1ef/test-curl-7551-140020560225424/0.log" Jul 06 19:50:18.496875 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:18.496846 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g89ss\" (UniqueName: \"kubernetes.io/projected/8a8ff464-0dd2-4a74-95ee-4813783ba1ef-kube-api-access-g89ss\") pod \"8a8ff464-0dd2-4a74-95ee-4813783ba1ef\" (UID: \"8a8ff464-0dd2-4a74-95ee-4813783ba1ef\") " Jul 06 19:50:18.499001 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:18.498978 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a8ff464-0dd2-4a74-95ee-4813783ba1ef-kube-api-access-g89ss" (OuterVolumeSpecName: "kube-api-access-g89ss") pod "8a8ff464-0dd2-4a74-95ee-4813783ba1ef" (UID: "8a8ff464-0dd2-4a74-95ee-4813783ba1ef"). InnerVolumeSpecName "kube-api-access-g89ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:50:18.506124 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:18.506099 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7551-140020560225424"] Jul 06 19:50:18.510521 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:18.510500 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7551-140020560225424"] Jul 06 19:50:18.597529 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:18.597456 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g89ss\" (UniqueName: \"kubernetes.io/projected/8a8ff464-0dd2-4a74-95ee-4813783ba1ef-kube-api-access-g89ss\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:50:18.641213 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:18.641185 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7551-140020560225536"] Jul 06 19:50:18.641559 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:18.641545 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a8ff464-0dd2-4a74-95ee-4813783ba1ef" containerName="test-curl-7551-140020560225424" Jul 06 19:50:18.641613 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:18.641561 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a8ff464-0dd2-4a74-95ee-4813783ba1ef" containerName="test-curl-7551-140020560225424" Jul 06 19:50:18.641649 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:18.641633 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a8ff464-0dd2-4a74-95ee-4813783ba1ef" containerName="test-curl-7551-140020560225424" Jul 06 19:50:18.644510 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:18.644495 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560225536" Jul 06 19:50:18.657134 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:18.657114 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7551-140020560225536"] Jul 06 19:50:18.698753 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:18.698730 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftld4\" (UniqueName: \"kubernetes.io/projected/4fb8eeda-e81d-42fe-9284-13c050efcbc4-kube-api-access-ftld4\") pod \"test-curl-7551-140020560225536\" (UID: \"4fb8eeda-e81d-42fe-9284-13c050efcbc4\") " pod="opendatahub/test-curl-7551-140020560225536" Jul 06 19:50:18.799821 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:18.799779 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftld4\" (UniqueName: \"kubernetes.io/projected/4fb8eeda-e81d-42fe-9284-13c050efcbc4-kube-api-access-ftld4\") pod \"test-curl-7551-140020560225536\" (UID: \"4fb8eeda-e81d-42fe-9284-13c050efcbc4\") " pod="opendatahub/test-curl-7551-140020560225536" Jul 06 19:50:18.809307 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:18.809276 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftld4\" (UniqueName: \"kubernetes.io/projected/4fb8eeda-e81d-42fe-9284-13c050efcbc4-kube-api-access-ftld4\") pod \"test-curl-7551-140020560225536\" (UID: \"4fb8eeda-e81d-42fe-9284-13c050efcbc4\") " pod="opendatahub/test-curl-7551-140020560225536" Jul 06 19:50:18.953202 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:18.953123 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560225536" Jul 06 19:50:19.088223 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:19.088097 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7551-140020560225536"] Jul 06 19:50:19.091018 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:50:19.090987 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fb8eeda_e81d_42fe_9284_13c050efcbc4.slice/crio-469d40ad98c80a303761439a1c7ef49c4b6664f7c823233c50d6035a878b1159 WatchSource:0}: Error finding container 469d40ad98c80a303761439a1c7ef49c4b6664f7c823233c50d6035a878b1159: Status 404 returned error can't find the container with id 469d40ad98c80a303761439a1c7ef49c4b6664f7c823233c50d6035a878b1159 Jul 06 19:50:19.283302 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:19.283271 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a8ff464-0dd2-4a74-95ee-4813783ba1ef" path="/var/lib/kubelet/pods/8a8ff464-0dd2-4a74-95ee-4813783ba1ef/volumes" Jul 06 19:50:19.308450 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:19.308398 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560225424" Jul 06 19:50:19.308602 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:19.308457 2553 scope.go:117] "RemoveContainer" containerID="2e6e7a8737f28ab00e12f0f0088d848df7124baa872b2b0553830562552cc058" Jul 06 19:50:19.310000 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:19.309975 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7551-140020560225536" event={"ID":"4fb8eeda-e81d-42fe-9284-13c050efcbc4","Type":"ContainerStarted","Data":"469d40ad98c80a303761439a1c7ef49c4b6664f7c823233c50d6035a878b1159"} Jul 06 19:50:20.317024 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:20.316991 2553 generic.go:356] "Generic (PLEG): container finished" podID="4fb8eeda-e81d-42fe-9284-13c050efcbc4" containerID="09db829f507746c97e029b39c012712cf3dcb9951d848e0c7ac34bd690677ef8" exitCode=0 Jul 06 19:50:20.317435 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:20.317085 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7551-140020560225536" event={"ID":"4fb8eeda-e81d-42fe-9284-13c050efcbc4","Type":"ContainerDied","Data":"09db829f507746c97e029b39c012712cf3dcb9951d848e0c7ac34bd690677ef8"} Jul 06 19:50:20.776948 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:20.776915 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:50:21.459781 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:21.459758 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560225536" Jul 06 19:50:21.482492 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:21.482468 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7551-140020560225536_4fb8eeda-e81d-42fe-9284-13c050efcbc4/test-curl-7551-140020560225536/0.log" Jul 06 19:50:21.515634 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:21.515600 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7551-140020560225536"] Jul 06 19:50:21.519798 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:21.519777 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7551-140020560225536"] Jul 06 19:50:21.529082 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:21.529059 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftld4\" (UniqueName: \"kubernetes.io/projected/4fb8eeda-e81d-42fe-9284-13c050efcbc4-kube-api-access-ftld4\") pod \"4fb8eeda-e81d-42fe-9284-13c050efcbc4\" (UID: \"4fb8eeda-e81d-42fe-9284-13c050efcbc4\") " Jul 06 19:50:21.531183 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:21.531157 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb8eeda-e81d-42fe-9284-13c050efcbc4-kube-api-access-ftld4" (OuterVolumeSpecName: "kube-api-access-ftld4") pod "4fb8eeda-e81d-42fe-9284-13c050efcbc4" (UID: "4fb8eeda-e81d-42fe-9284-13c050efcbc4"). InnerVolumeSpecName "kube-api-access-ftld4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:50:21.630364 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:21.630280 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftld4\" (UniqueName: \"kubernetes.io/projected/4fb8eeda-e81d-42fe-9284-13c050efcbc4-kube-api-access-ftld4\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:50:21.667472 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:21.667440 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7551-140020560224752"] Jul 06 19:50:21.667825 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:21.667812 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fb8eeda-e81d-42fe-9284-13c050efcbc4" containerName="test-curl-7551-140020560225536" Jul 06 19:50:21.667883 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:21.667827 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fb8eeda-e81d-42fe-9284-13c050efcbc4" containerName="test-curl-7551-140020560225536" Jul 06 19:50:21.667915 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:21.667890 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fb8eeda-e81d-42fe-9284-13c050efcbc4" containerName="test-curl-7551-140020560225536" Jul 06 19:50:21.670868 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:21.670850 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560224752" Jul 06 19:50:21.689304 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:21.689284 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7551-140020560224752"] Jul 06 19:50:21.730961 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:21.730930 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhntk\" (UniqueName: \"kubernetes.io/projected/8a73d4f0-d088-4d45-88a4-5dbd0f08865c-kube-api-access-vhntk\") pod \"test-curl-7551-140020560224752\" (UID: \"8a73d4f0-d088-4d45-88a4-5dbd0f08865c\") " pod="opendatahub/test-curl-7551-140020560224752" Jul 06 19:50:21.831586 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:21.831557 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhntk\" (UniqueName: \"kubernetes.io/projected/8a73d4f0-d088-4d45-88a4-5dbd0f08865c-kube-api-access-vhntk\") pod \"test-curl-7551-140020560224752\" (UID: \"8a73d4f0-d088-4d45-88a4-5dbd0f08865c\") " pod="opendatahub/test-curl-7551-140020560224752" Jul 06 19:50:21.844701 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:21.844672 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhntk\" (UniqueName: \"kubernetes.io/projected/8a73d4f0-d088-4d45-88a4-5dbd0f08865c-kube-api-access-vhntk\") pod \"test-curl-7551-140020560224752\" (UID: \"8a73d4f0-d088-4d45-88a4-5dbd0f08865c\") " pod="opendatahub/test-curl-7551-140020560224752" Jul 06 19:50:21.980333 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:21.980299 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560224752" Jul 06 19:50:22.119208 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:22.119102 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7551-140020560224752"] Jul 06 19:50:22.121747 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:50:22.121718 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a73d4f0_d088_4d45_88a4_5dbd0f08865c.slice/crio-004a4cf1f3ef747b98a50acf44364cfd2630a28defa890b91cb2e4b8cb7922da WatchSource:0}: Error finding container 004a4cf1f3ef747b98a50acf44364cfd2630a28defa890b91cb2e4b8cb7922da: Status 404 returned error can't find the container with id 004a4cf1f3ef747b98a50acf44364cfd2630a28defa890b91cb2e4b8cb7922da Jul 06 19:50:22.327738 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:22.327660 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560225536" Jul 06 19:50:22.327895 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:22.327659 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="469d40ad98c80a303761439a1c7ef49c4b6664f7c823233c50d6035a878b1159" Jul 06 19:50:22.328953 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:22.328926 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7551-140020560224752" event={"ID":"8a73d4f0-d088-4d45-88a4-5dbd0f08865c","Type":"ContainerStarted","Data":"004a4cf1f3ef747b98a50acf44364cfd2630a28defa890b91cb2e4b8cb7922da"} Jul 06 19:50:22.338613 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:50:22.338590 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7551-140020560225536\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="4fb8eeda-e81d-42fe-9284-13c050efcbc4" pod="opendatahub/test-curl-7551-140020560225536" Jul 06 19:50:23.289607 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:23.289569 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fb8eeda-e81d-42fe-9284-13c050efcbc4" path="/var/lib/kubelet/pods/4fb8eeda-e81d-42fe-9284-13c050efcbc4/volumes" Jul 06 19:50:23.335029 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:23.335000 2553 generic.go:356] "Generic (PLEG): container finished" podID="8a73d4f0-d088-4d45-88a4-5dbd0f08865c" containerID="4006e16674ecf2ff3eb6a959d7482abc57ef206c02022201275f6720927fce39" exitCode=0 Jul 06 19:50:23.335203 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:23.335029 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7551-140020560224752" event={"ID":"8a73d4f0-d088-4d45-88a4-5dbd0f08865c","Type":"ContainerDied","Data":"4006e16674ecf2ff3eb6a959d7482abc57ef206c02022201275f6720927fce39"} Jul 06 19:50:24.476391 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:24.476362 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560224752" Jul 06 19:50:24.496440 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:24.496394 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7551-140020560224752_8a73d4f0-d088-4d45-88a4-5dbd0f08865c/test-curl-7551-140020560224752/0.log" Jul 06 19:50:24.519008 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:24.518970 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7551-140020560224752"] Jul 06 19:50:24.521153 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:24.521132 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7551-140020560224752"] Jul 06 19:50:24.558350 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:24.558320 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhntk\" (UniqueName: \"kubernetes.io/projected/8a73d4f0-d088-4d45-88a4-5dbd0f08865c-kube-api-access-vhntk\") pod \"8a73d4f0-d088-4d45-88a4-5dbd0f08865c\" (UID: \"8a73d4f0-d088-4d45-88a4-5dbd0f08865c\") " Jul 06 19:50:24.560590 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:24.560568 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a73d4f0-d088-4d45-88a4-5dbd0f08865c-kube-api-access-vhntk" (OuterVolumeSpecName: "kube-api-access-vhntk") pod "8a73d4f0-d088-4d45-88a4-5dbd0f08865c" (UID: "8a73d4f0-d088-4d45-88a4-5dbd0f08865c"). InnerVolumeSpecName "kube-api-access-vhntk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:50:24.659046 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:24.658963 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhntk\" (UniqueName: \"kubernetes.io/projected/8a73d4f0-d088-4d45-88a4-5dbd0f08865c-kube-api-access-vhntk\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:50:24.690166 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:24.690136 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7551-140020560245360"] Jul 06 19:50:24.690530 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:24.690516 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a73d4f0-d088-4d45-88a4-5dbd0f08865c" containerName="test-curl-7551-140020560224752" Jul 06 19:50:24.690571 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:24.690532 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a73d4f0-d088-4d45-88a4-5dbd0f08865c" containerName="test-curl-7551-140020560224752" Jul 06 19:50:24.690611 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:24.690602 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a73d4f0-d088-4d45-88a4-5dbd0f08865c" containerName="test-curl-7551-140020560224752" Jul 06 19:50:24.693580 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:24.693565 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560245360" Jul 06 19:50:24.700009 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:24.699990 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7551-140020560245360"] Jul 06 19:50:24.760437 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:24.760394 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bg5c\" (UniqueName: \"kubernetes.io/projected/aa9a3b37-98e2-493e-bc66-b4330592af06-kube-api-access-9bg5c\") pod \"test-curl-7551-140020560245360\" (UID: \"aa9a3b37-98e2-493e-bc66-b4330592af06\") " pod="opendatahub/test-curl-7551-140020560245360" Jul 06 19:50:24.861492 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:24.861457 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bg5c\" (UniqueName: \"kubernetes.io/projected/aa9a3b37-98e2-493e-bc66-b4330592af06-kube-api-access-9bg5c\") pod \"test-curl-7551-140020560245360\" (UID: \"aa9a3b37-98e2-493e-bc66-b4330592af06\") " pod="opendatahub/test-curl-7551-140020560245360" Jul 06 19:50:24.870468 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:24.870446 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bg5c\" (UniqueName: \"kubernetes.io/projected/aa9a3b37-98e2-493e-bc66-b4330592af06-kube-api-access-9bg5c\") pod \"test-curl-7551-140020560245360\" (UID: \"aa9a3b37-98e2-493e-bc66-b4330592af06\") " pod="opendatahub/test-curl-7551-140020560245360" Jul 06 19:50:25.003851 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:25.003814 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560245360" Jul 06 19:50:25.139536 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:25.139508 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7551-140020560245360"] Jul 06 19:50:25.141522 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:50:25.141495 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa9a3b37_98e2_493e_bc66_b4330592af06.slice/crio-c4957a5a0ed0d2ac4f8a06419f0b9eea954d6a882ee51822c221252dc43f4c61 WatchSource:0}: Error finding container c4957a5a0ed0d2ac4f8a06419f0b9eea954d6a882ee51822c221252dc43f4c61: Status 404 returned error can't find the container with id c4957a5a0ed0d2ac4f8a06419f0b9eea954d6a882ee51822c221252dc43f4c61 Jul 06 19:50:25.282734 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:25.282661 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a73d4f0-d088-4d45-88a4-5dbd0f08865c" path="/var/lib/kubelet/pods/8a73d4f0-d088-4d45-88a4-5dbd0f08865c/volumes" Jul 06 19:50:25.345291 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:25.345253 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7551-140020560245360" event={"ID":"aa9a3b37-98e2-493e-bc66-b4330592af06","Type":"ContainerStarted","Data":"c4957a5a0ed0d2ac4f8a06419f0b9eea954d6a882ee51822c221252dc43f4c61"} Jul 06 19:50:25.346495 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:25.346476 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560224752" Jul 06 19:50:25.346594 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:25.346519 2553 scope.go:117] "RemoveContainer" containerID="4006e16674ecf2ff3eb6a959d7482abc57ef206c02022201275f6720927fce39" Jul 06 19:50:26.353363 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:26.353329 2553 generic.go:356] "Generic (PLEG): container finished" podID="aa9a3b37-98e2-493e-bc66-b4330592af06" containerID="ebf419e1cae453d3abea0237e3b43fa62f3cd62fa355ef04056e10043bd52779" exitCode=0 Jul 06 19:50:26.353821 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:26.353434 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7551-140020560245360" event={"ID":"aa9a3b37-98e2-493e-bc66-b4330592af06","Type":"ContainerDied","Data":"ebf419e1cae453d3abea0237e3b43fa62f3cd62fa355ef04056e10043bd52779"} Jul 06 19:50:27.530734 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:27.530711 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560245360" Jul 06 19:50:27.553528 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:27.553500 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7551-140020560245360_aa9a3b37-98e2-493e-bc66-b4330592af06/test-curl-7551-140020560245360/0.log" Jul 06 19:50:27.579142 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:27.579112 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7551-140020560245360"] Jul 06 19:50:27.583094 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:27.583069 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7551-140020560245360"] Jul 06 19:50:27.587889 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:27.587861 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bg5c\" (UniqueName: \"kubernetes.io/projected/aa9a3b37-98e2-493e-bc66-b4330592af06-kube-api-access-9bg5c\") pod \"aa9a3b37-98e2-493e-bc66-b4330592af06\" (UID: \"aa9a3b37-98e2-493e-bc66-b4330592af06\") " Jul 06 19:50:27.590068 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:27.590041 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9a3b37-98e2-493e-bc66-b4330592af06-kube-api-access-9bg5c" (OuterVolumeSpecName: "kube-api-access-9bg5c") pod "aa9a3b37-98e2-493e-bc66-b4330592af06" (UID: "aa9a3b37-98e2-493e-bc66-b4330592af06"). InnerVolumeSpecName "kube-api-access-9bg5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:50:27.689531 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:27.689453 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bg5c\" (UniqueName: \"kubernetes.io/projected/aa9a3b37-98e2-493e-bc66-b4330592af06-kube-api-access-9bg5c\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:50:27.707475 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:27.707442 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7551-140020564180400"] Jul 06 19:50:27.707836 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:27.707823 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa9a3b37-98e2-493e-bc66-b4330592af06" containerName="test-curl-7551-140020560245360" Jul 06 19:50:27.707891 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:27.707838 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa9a3b37-98e2-493e-bc66-b4330592af06" containerName="test-curl-7551-140020560245360" Jul 06 19:50:27.707939 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:27.707898 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa9a3b37-98e2-493e-bc66-b4330592af06" containerName="test-curl-7551-140020560245360" Jul 06 19:50:27.710796 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:27.710780 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020564180400" Jul 06 19:50:27.716934 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:27.716906 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7551-140020564180400"] Jul 06 19:50:27.790589 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:27.790558 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrznz\" (UniqueName: \"kubernetes.io/projected/ef6299ab-e654-4370-80fc-7ece220c31d0-kube-api-access-zrznz\") pod \"test-curl-7551-140020564180400\" (UID: \"ef6299ab-e654-4370-80fc-7ece220c31d0\") " pod="opendatahub/test-curl-7551-140020564180400" Jul 06 19:50:27.891987 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:27.891952 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrznz\" (UniqueName: \"kubernetes.io/projected/ef6299ab-e654-4370-80fc-7ece220c31d0-kube-api-access-zrznz\") pod \"test-curl-7551-140020564180400\" (UID: \"ef6299ab-e654-4370-80fc-7ece220c31d0\") " pod="opendatahub/test-curl-7551-140020564180400" Jul 06 19:50:27.901080 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:27.901050 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrznz\" (UniqueName: \"kubernetes.io/projected/ef6299ab-e654-4370-80fc-7ece220c31d0-kube-api-access-zrznz\") pod \"test-curl-7551-140020564180400\" (UID: \"ef6299ab-e654-4370-80fc-7ece220c31d0\") " pod="opendatahub/test-curl-7551-140020564180400" Jul 06 19:50:28.021249 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:28.021219 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020564180400" Jul 06 19:50:28.160486 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:28.160461 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7551-140020564180400"] Jul 06 19:50:28.163149 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:50:28.163108 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef6299ab_e654_4370_80fc_7ece220c31d0.slice/crio-b91075b9a23f90669083e01ceedb3a1eae8c7bb6c9cc26679204ce5de1a9dc28 WatchSource:0}: Error finding container b91075b9a23f90669083e01ceedb3a1eae8c7bb6c9cc26679204ce5de1a9dc28: Status 404 returned error can't find the container with id b91075b9a23f90669083e01ceedb3a1eae8c7bb6c9cc26679204ce5de1a9dc28 Jul 06 19:50:28.371967 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:28.371883 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4957a5a0ed0d2ac4f8a06419f0b9eea954d6a882ee51822c221252dc43f4c61" Jul 06 19:50:28.371967 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:28.371909 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020560245360" Jul 06 19:50:28.373256 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:28.373224 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7551-140020564180400" event={"ID":"ef6299ab-e654-4370-80fc-7ece220c31d0","Type":"ContainerStarted","Data":"b91075b9a23f90669083e01ceedb3a1eae8c7bb6c9cc26679204ce5de1a9dc28"} Jul 06 19:50:28.382822 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:50:28.382795 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7551-140020560245360\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="aa9a3b37-98e2-493e-bc66-b4330592af06" pod="opendatahub/test-curl-7551-140020560245360" Jul 06 19:50:29.281295 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:50:29.281254 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7551-140020560245360\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="aa9a3b37-98e2-493e-bc66-b4330592af06" pod="opendatahub/test-curl-7551-140020560245360" Jul 06 19:50:29.283505 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:29.283482 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa9a3b37-98e2-493e-bc66-b4330592af06" path="/var/lib/kubelet/pods/aa9a3b37-98e2-493e-bc66-b4330592af06/volumes" Jul 06 19:50:29.379358 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:29.379329 2553 generic.go:356] "Generic (PLEG): container finished" podID="ef6299ab-e654-4370-80fc-7ece220c31d0" containerID="7263f69821d3067a609cfd7c5e26584a1ee2237215b951dc57eb5f01fa284982" exitCode=0 Jul 06 19:50:29.379546 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:29.379436 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7551-140020564180400" event={"ID":"ef6299ab-e654-4370-80fc-7ece220c31d0","Type":"ContainerDied","Data":"7263f69821d3067a609cfd7c5e26584a1ee2237215b951dc57eb5f01fa284982"} Jul 06 19:50:30.523583 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:30.523561 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020564180400" Jul 06 19:50:30.545936 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:30.545911 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7551-140020564180400_ef6299ab-e654-4370-80fc-7ece220c31d0/test-curl-7551-140020564180400/0.log" Jul 06 19:50:30.568475 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:30.568452 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7551-140020564180400"] Jul 06 19:50:30.571567 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:30.571543 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7551-140020564180400"] Jul 06 19:50:30.615250 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:30.615223 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrznz\" (UniqueName: \"kubernetes.io/projected/ef6299ab-e654-4370-80fc-7ece220c31d0-kube-api-access-zrznz\") pod \"ef6299ab-e654-4370-80fc-7ece220c31d0\" (UID: \"ef6299ab-e654-4370-80fc-7ece220c31d0\") " Jul 06 19:50:30.617384 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:30.617357 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6299ab-e654-4370-80fc-7ece220c31d0-kube-api-access-zrznz" (OuterVolumeSpecName: "kube-api-access-zrznz") pod "ef6299ab-e654-4370-80fc-7ece220c31d0" (UID: "ef6299ab-e654-4370-80fc-7ece220c31d0"). InnerVolumeSpecName "kube-api-access-zrznz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:50:30.693675 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:30.693616 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7551-140020564180256"] Jul 06 19:50:30.693986 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:30.693973 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef6299ab-e654-4370-80fc-7ece220c31d0" containerName="test-curl-7551-140020564180400" Jul 06 19:50:30.694030 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:30.693987 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef6299ab-e654-4370-80fc-7ece220c31d0" containerName="test-curl-7551-140020564180400" Jul 06 19:50:30.694066 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:30.694053 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef6299ab-e654-4370-80fc-7ece220c31d0" containerName="test-curl-7551-140020564180400" Jul 06 19:50:30.697221 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:30.697205 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020564180256" Jul 06 19:50:30.703191 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:30.703170 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7551-140020564180256"] Jul 06 19:50:30.716489 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:30.716469 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrznz\" (UniqueName: \"kubernetes.io/projected/ef6299ab-e654-4370-80fc-7ece220c31d0-kube-api-access-zrznz\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:50:30.817379 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:30.817349 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-472w2\" (UniqueName: \"kubernetes.io/projected/97f61d9a-8236-41d9-bd83-1a803b92ff5c-kube-api-access-472w2\") pod \"test-curl-7551-140020564180256\" (UID: \"97f61d9a-8236-41d9-bd83-1a803b92ff5c\") " pod="opendatahub/test-curl-7551-140020564180256" Jul 06 19:50:30.917934 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:30.917905 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-472w2\" (UniqueName: \"kubernetes.io/projected/97f61d9a-8236-41d9-bd83-1a803b92ff5c-kube-api-access-472w2\") pod \"test-curl-7551-140020564180256\" (UID: \"97f61d9a-8236-41d9-bd83-1a803b92ff5c\") " pod="opendatahub/test-curl-7551-140020564180256" Jul 06 19:50:30.926298 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:30.926271 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-472w2\" (UniqueName: \"kubernetes.io/projected/97f61d9a-8236-41d9-bd83-1a803b92ff5c-kube-api-access-472w2\") pod \"test-curl-7551-140020564180256\" (UID: \"97f61d9a-8236-41d9-bd83-1a803b92ff5c\") " pod="opendatahub/test-curl-7551-140020564180256" Jul 06 19:50:31.006280 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:31.006251 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020564180256" Jul 06 19:50:31.139864 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:31.139841 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7551-140020564180256"] Jul 06 19:50:31.142182 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:50:31.142158 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97f61d9a_8236_41d9_bd83_1a803b92ff5c.slice/crio-e72a9059c96aaf5fb19a422b4b2384c848260dbc176146dfa1c59eecbcca9117 WatchSource:0}: Error finding container e72a9059c96aaf5fb19a422b4b2384c848260dbc176146dfa1c59eecbcca9117: Status 404 returned error can't find the container with id e72a9059c96aaf5fb19a422b4b2384c848260dbc176146dfa1c59eecbcca9117 Jul 06 19:50:31.283133 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:31.283064 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef6299ab-e654-4370-80fc-7ece220c31d0" path="/var/lib/kubelet/pods/ef6299ab-e654-4370-80fc-7ece220c31d0/volumes" Jul 06 19:50:31.390745 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:31.390711 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7551-140020564180256" event={"ID":"97f61d9a-8236-41d9-bd83-1a803b92ff5c","Type":"ContainerStarted","Data":"e72a9059c96aaf5fb19a422b4b2384c848260dbc176146dfa1c59eecbcca9117"} Jul 06 19:50:31.392004 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:31.391980 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020564180400" Jul 06 19:50:31.392134 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:31.392025 2553 scope.go:117] "RemoveContainer" containerID="7263f69821d3067a609cfd7c5e26584a1ee2237215b951dc57eb5f01fa284982" Jul 06 19:50:32.398685 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:32.398651 2553 generic.go:356] "Generic (PLEG): container finished" podID="97f61d9a-8236-41d9-bd83-1a803b92ff5c" containerID="6a0d8f0fe215d0bdf02d5507b0c64e847524a6c31544dc6a6d2c67685ac1d9c0" exitCode=0 Jul 06 19:50:32.399047 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:32.398732 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7551-140020564180256" event={"ID":"97f61d9a-8236-41d9-bd83-1a803b92ff5c","Type":"ContainerDied","Data":"6a0d8f0fe215d0bdf02d5507b0c64e847524a6c31544dc6a6d2c67685ac1d9c0"} Jul 06 19:50:33.480897 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:33.480870 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:50:33.550115 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:33.550091 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020564180256" Jul 06 19:50:33.580546 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:33.580524 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7551-140020564180256_97f61d9a-8236-41d9-bd83-1a803b92ff5c/test-curl-7551-140020564180256/0.log" Jul 06 19:50:33.611873 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:33.611842 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7551-140020564180256"] Jul 06 19:50:33.613646 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:33.613622 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7551-140020564180256"] Jul 06 19:50:33.642138 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:33.642112 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-472w2\" (UniqueName: \"kubernetes.io/projected/97f61d9a-8236-41d9-bd83-1a803b92ff5c-kube-api-access-472w2\") pod \"97f61d9a-8236-41d9-bd83-1a803b92ff5c\" (UID: \"97f61d9a-8236-41d9-bd83-1a803b92ff5c\") " Jul 06 19:50:33.644305 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:33.644282 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f61d9a-8236-41d9-bd83-1a803b92ff5c-kube-api-access-472w2" (OuterVolumeSpecName: "kube-api-access-472w2") pod "97f61d9a-8236-41d9-bd83-1a803b92ff5c" (UID: "97f61d9a-8236-41d9-bd83-1a803b92ff5c"). InnerVolumeSpecName "kube-api-access-472w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:50:33.743189 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:33.743158 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-472w2\" (UniqueName: \"kubernetes.io/projected/97f61d9a-8236-41d9-bd83-1a803b92ff5c-kube-api-access-472w2\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:50:33.749108 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:33.749084 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7551-140020564179248"] Jul 06 19:50:33.749491 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:33.749477 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97f61d9a-8236-41d9-bd83-1a803b92ff5c" containerName="test-curl-7551-140020564180256" Jul 06 19:50:33.749536 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:33.749493 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="97f61d9a-8236-41d9-bd83-1a803b92ff5c" containerName="test-curl-7551-140020564180256" Jul 06 19:50:33.749581 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:33.749572 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="97f61d9a-8236-41d9-bd83-1a803b92ff5c" containerName="test-curl-7551-140020564180256" Jul 06 19:50:33.752721 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:33.752706 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020564179248" Jul 06 19:50:33.760740 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:33.760719 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7551-140020564179248"] Jul 06 19:50:33.844663 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:33.844629 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gc89\" (UniqueName: \"kubernetes.io/projected/575f9932-b09c-4c79-92b6-f454073fd982-kube-api-access-7gc89\") pod \"test-curl-7551-140020564179248\" (UID: \"575f9932-b09c-4c79-92b6-f454073fd982\") " pod="opendatahub/test-curl-7551-140020564179248" Jul 06 19:50:33.945563 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:33.945531 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gc89\" (UniqueName: \"kubernetes.io/projected/575f9932-b09c-4c79-92b6-f454073fd982-kube-api-access-7gc89\") pod \"test-curl-7551-140020564179248\" (UID: \"575f9932-b09c-4c79-92b6-f454073fd982\") " pod="opendatahub/test-curl-7551-140020564179248" Jul 06 19:50:33.954588 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:33.954564 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gc89\" (UniqueName: \"kubernetes.io/projected/575f9932-b09c-4c79-92b6-f454073fd982-kube-api-access-7gc89\") pod \"test-curl-7551-140020564179248\" (UID: \"575f9932-b09c-4c79-92b6-f454073fd982\") " pod="opendatahub/test-curl-7551-140020564179248" Jul 06 19:50:34.062069 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:34.061986 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020564179248" Jul 06 19:50:34.201196 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:34.201170 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7551-140020564179248"] Jul 06 19:50:34.203704 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:50:34.203674 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod575f9932_b09c_4c79_92b6_f454073fd982.slice/crio-7a46307b9efbd332886e6d0e7c595d0a74ad58737b9d082e9e5cd0e5e4be727c WatchSource:0}: Error finding container 7a46307b9efbd332886e6d0e7c595d0a74ad58737b9d082e9e5cd0e5e4be727c: Status 404 returned error can't find the container with id 7a46307b9efbd332886e6d0e7c595d0a74ad58737b9d082e9e5cd0e5e4be727c Jul 06 19:50:34.413293 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:34.413207 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e72a9059c96aaf5fb19a422b4b2384c848260dbc176146dfa1c59eecbcca9117" Jul 06 19:50:34.413293 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:34.413205 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020564180256" Jul 06 19:50:34.414339 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:34.414317 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7551-140020564179248" event={"ID":"575f9932-b09c-4c79-92b6-f454073fd982","Type":"ContainerStarted","Data":"7a46307b9efbd332886e6d0e7c595d0a74ad58737b9d082e9e5cd0e5e4be727c"} Jul 06 19:50:34.423561 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:50:34.423536 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7551-140020564180256\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="97f61d9a-8236-41d9-bd83-1a803b92ff5c" pod="opendatahub/test-curl-7551-140020564180256" Jul 06 19:50:35.283670 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:35.283637 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97f61d9a-8236-41d9-bd83-1a803b92ff5c" path="/var/lib/kubelet/pods/97f61d9a-8236-41d9-bd83-1a803b92ff5c/volumes" Jul 06 19:50:35.421695 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:35.421655 2553 generic.go:356] "Generic (PLEG): container finished" podID="575f9932-b09c-4c79-92b6-f454073fd982" containerID="33d351004b8338a6ad30d9da14cfa970c022ff8f6ed5ed49142e12997770c27b" exitCode=0 Jul 06 19:50:35.421859 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:35.421734 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7551-140020564179248" event={"ID":"575f9932-b09c-4c79-92b6-f454073fd982","Type":"ContainerDied","Data":"33d351004b8338a6ad30d9da14cfa970c022ff8f6ed5ed49142e12997770c27b"} Jul 06 19:50:36.559276 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:36.559249 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020564179248" Jul 06 19:50:36.581721 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:36.581697 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7551-140020564179248_575f9932-b09c-4c79-92b6-f454073fd982/test-curl-7551-140020564179248/0.log" Jul 06 19:50:36.608061 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:36.608030 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7551-140020564179248"] Jul 06 19:50:36.612556 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:36.612534 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7551-140020564179248"] Jul 06 19:50:36.670587 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:36.670562 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gc89\" (UniqueName: \"kubernetes.io/projected/575f9932-b09c-4c79-92b6-f454073fd982-kube-api-access-7gc89\") pod \"575f9932-b09c-4c79-92b6-f454073fd982\" (UID: \"575f9932-b09c-4c79-92b6-f454073fd982\") " Jul 06 19:50:36.672668 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:36.672650 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575f9932-b09c-4c79-92b6-f454073fd982-kube-api-access-7gc89" (OuterVolumeSpecName: "kube-api-access-7gc89") pod "575f9932-b09c-4c79-92b6-f454073fd982" (UID: "575f9932-b09c-4c79-92b6-f454073fd982"). InnerVolumeSpecName "kube-api-access-7gc89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:50:36.747645 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:36.747617 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7551-140020564180688"] Jul 06 19:50:36.747979 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:36.747967 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="575f9932-b09c-4c79-92b6-f454073fd982" containerName="test-curl-7551-140020564179248" Jul 06 19:50:36.748020 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:36.747981 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="575f9932-b09c-4c79-92b6-f454073fd982" containerName="test-curl-7551-140020564179248" Jul 06 19:50:36.748063 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:36.748054 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="575f9932-b09c-4c79-92b6-f454073fd982" containerName="test-curl-7551-140020564179248" Jul 06 19:50:36.751101 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:36.751086 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020564180688" Jul 06 19:50:36.755260 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:36.755240 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7551-140020564180688"] Jul 06 19:50:36.771172 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:36.771151 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gc89\" (UniqueName: \"kubernetes.io/projected/575f9932-b09c-4c79-92b6-f454073fd982-kube-api-access-7gc89\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:50:36.872099 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:36.872071 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-splkr\" (UniqueName: \"kubernetes.io/projected/f896f84e-fbab-4513-ac5f-5a613009267c-kube-api-access-splkr\") pod \"test-curl-7551-140020564180688\" (UID: \"f896f84e-fbab-4513-ac5f-5a613009267c\") " pod="opendatahub/test-curl-7551-140020564180688" Jul 06 19:50:36.973403 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:36.973373 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-splkr\" (UniqueName: \"kubernetes.io/projected/f896f84e-fbab-4513-ac5f-5a613009267c-kube-api-access-splkr\") pod \"test-curl-7551-140020564180688\" (UID: \"f896f84e-fbab-4513-ac5f-5a613009267c\") " pod="opendatahub/test-curl-7551-140020564180688" Jul 06 19:50:36.983572 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:36.983547 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-splkr\" (UniqueName: \"kubernetes.io/projected/f896f84e-fbab-4513-ac5f-5a613009267c-kube-api-access-splkr\") pod \"test-curl-7551-140020564180688\" (UID: \"f896f84e-fbab-4513-ac5f-5a613009267c\") " pod="opendatahub/test-curl-7551-140020564180688" Jul 06 19:50:37.062146 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:37.062119 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020564180688" Jul 06 19:50:37.196577 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:37.196549 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7551-140020564180688"] Jul 06 19:50:37.199117 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:50:37.199089 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf896f84e_fbab_4513_ac5f_5a613009267c.slice/crio-eb8c836be53e1a0af7dbf7ee2da169aa49fdc5d02c4155464a9c4864114faa85 WatchSource:0}: Error finding container eb8c836be53e1a0af7dbf7ee2da169aa49fdc5d02c4155464a9c4864114faa85: Status 404 returned error can't find the container with id eb8c836be53e1a0af7dbf7ee2da169aa49fdc5d02c4155464a9c4864114faa85 Jul 06 19:50:37.282517 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:37.282486 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="575f9932-b09c-4c79-92b6-f454073fd982" path="/var/lib/kubelet/pods/575f9932-b09c-4c79-92b6-f454073fd982/volumes" Jul 06 19:50:37.432299 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:37.432262 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7551-140020564180688" event={"ID":"f896f84e-fbab-4513-ac5f-5a613009267c","Type":"ContainerStarted","Data":"eb8c836be53e1a0af7dbf7ee2da169aa49fdc5d02c4155464a9c4864114faa85"} Jul 06 19:50:37.433536 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:37.433515 2553 scope.go:117] "RemoveContainer" containerID="33d351004b8338a6ad30d9da14cfa970c022ff8f6ed5ed49142e12997770c27b" Jul 06 19:50:37.433678 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:37.433565 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020564179248" Jul 06 19:50:38.439903 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:38.439871 2553 generic.go:356] "Generic (PLEG): container finished" podID="f896f84e-fbab-4513-ac5f-5a613009267c" containerID="ccfd177a8034cd61eee509f163016c2473fb040f21599dc1ef89304afe523b6c" exitCode=0 Jul 06 19:50:38.440437 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:38.439966 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7551-140020564180688" event={"ID":"f896f84e-fbab-4513-ac5f-5a613009267c","Type":"ContainerDied","Data":"ccfd177a8034cd61eee509f163016c2473fb040f21599dc1ef89304afe523b6c"} Jul 06 19:50:39.587386 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:39.587365 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020564180688" Jul 06 19:50:39.607636 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:39.607607 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7551-140020564180688_f896f84e-fbab-4513-ac5f-5a613009267c/test-curl-7551-140020564180688/0.log" Jul 06 19:50:39.632490 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:39.632466 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7551-140020564180688"] Jul 06 19:50:39.639650 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:39.639629 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7551-140020564180688"] Jul 06 19:50:39.697696 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:39.697635 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-splkr\" (UniqueName: \"kubernetes.io/projected/f896f84e-fbab-4513-ac5f-5a613009267c-kube-api-access-splkr\") pod \"f896f84e-fbab-4513-ac5f-5a613009267c\" (UID: \"f896f84e-fbab-4513-ac5f-5a613009267c\") " Jul 06 19:50:39.699764 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:39.699741 2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f896f84e-fbab-4513-ac5f-5a613009267c-kube-api-access-splkr" (OuterVolumeSpecName: "kube-api-access-splkr") pod "f896f84e-fbab-4513-ac5f-5a613009267c" (UID: "f896f84e-fbab-4513-ac5f-5a613009267c"). InnerVolumeSpecName "kube-api-access-splkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:50:39.798903 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:39.798878 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-splkr\" (UniqueName: \"kubernetes.io/projected/f896f84e-fbab-4513-ac5f-5a613009267c-kube-api-access-splkr\") on node \"ip-10-0-141-237.ec2.internal\" DevicePath \"\"" Jul 06 19:50:40.452748 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:40.452713 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7551-140020564180688" Jul 06 19:50:40.452941 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:40.452769 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb8c836be53e1a0af7dbf7ee2da169aa49fdc5d02c4155464a9c4864114faa85" Jul 06 19:50:40.462013 ip-10-0-141-237 kubenswrapper[2553]: E0706 19:50:40.461987 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7551-140020564180688\" is forbidden: User \"system:node:ip-10-0-141-237.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-237.ec2.internal' and this object" podUID="f896f84e-fbab-4513-ac5f-5a613009267c" pod="opendatahub/test-curl-7551-140020564180688" Jul 06 19:50:41.282956 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:41.282922 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f896f84e-fbab-4513-ac5f-5a613009267c" path="/var/lib/kubelet/pods/f896f84e-fbab-4513-ac5f-5a613009267c/volumes" Jul 06 19:50:57.970713 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:50:57.970681 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:51:17.137578 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:17.137550 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-5c92ad-29722785-zn4xm"] Jul 06 19:51:17.153780 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:17.153757 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-5c92ad-29722785-zn4xm"] Jul 06 19:51:17.283157 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:17.283125 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="724e5c31-c548-449c-a376-43916cc5c0a2" path="/var/lib/kubelet/pods/724e5c31-c548-449c-a376-43916cc5c0a2/volumes" Jul 06 19:51:22.992569 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:22.992494 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xh6vt"] Jul 06 19:51:32.724044 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:32.724008 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gwbxn_ad5da93d-e1e5-4854-98fb-7d8f95143b13/manager/4.log" Jul 06 19:51:32.841868 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:32.841837 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-sq8dz_ba75546a-9e3d-4363-a68d-b194380ff105/manager/4.log" Jul 06 19:51:33.086485 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:33.086391 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722755-55kz6_d82ab975-83d9-4c4b-93ee-13c52fced6b3/cleanup/0.log" Jul 06 19:51:33.560554 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:33.560526 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-5psq2_d59bbf4a-f17c-43a9-ab25-d64b4b351819/server/0.log" Jul 06 19:51:33.817875 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:33.817784 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w_c9fa46fe-a84d-449b-a63b-47121b01ba82/manager/0.log" Jul 06 19:51:33.827125 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:33.827105 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w_c9fa46fe-a84d-449b-a63b-47121b01ba82/copy-manifests/0.log" Jul 06 19:51:35.684740 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:35.684708 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-kr5j9_2ab81f98-e81a-4b24-beae-8babf214f1fd/manager/3.log" Jul 06 19:51:35.803055 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:35.803027 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-wklwt_632d5de0-2e1b-4124-906e-21a096713b69/manager/4.log" Jul 06 19:51:35.918869 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:35.918842 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-vcvtg_ec841e47-d3ca-43bf-a062-361f912c5f27/kuadrant-console-plugin/0.log" Jul 06 19:51:36.037656 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:36.037627 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6tkch_5c49682e-63e9-4178-91ca-a2021cc916ed/registry-server/0.log" Jul 06 19:51:36.163879 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:36.163849 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-7w2nd_995c4fe8-11b1-4e0b-8190-e9c686f0a274/manager/0.log" Jul 06 19:51:36.275738 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:36.275711 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-xh6vt_18559278-46e1-4a51-a067-8a11b7414264/limitador/0.log" Jul 06 19:51:36.737630 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:36.737601 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4j46q6_6ea08144-1c62-42ec-8d15-3da05e0fa17b/istio-proxy/0.log" Jul 06 19:51:37.084051 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:37.083976 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c9545dd54-sjc64_9780aa2b-78ab-489e-be4b-cefc925b9087/kube-auth-proxy/0.log" Jul 06 19:51:37.199476 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:37.199447 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lcj97_6b30dece-56c5-458a-82b9-f799d264cb5a/istio-proxy/1.log" Jul 06 19:51:37.571062 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:37.571031 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7456bbb6dc-zfg4t_b78c33c0-d4f4-4a97-8621-90904852db08/router/0.log" Jul 06 19:51:38.048296 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:38.048271 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-q7njj_66840435-d9c5-4e2c-9890-c62a74662880/main/0.log" Jul 06 19:51:38.280205 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:38.280180 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbwfcfx_a76e45ae-863a-4358-985d-2295f154dd25/main/0.log" Jul 06 19:51:38.398350 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:38.398276 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-mpm2t_d60fae4e-c9ac-4baf-88ad-647922d26bd3/main/0.log" Jul 06 19:51:38.509752 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:38.509720 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-vhqbw_f80b76a2-b565-4c5f-b234-0db0d4ded27f/main/0.log" Jul 06 19:51:46.280403 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:46.280370 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v5xs7_f7d2aeda-aa6f-4865-ade0-f9aa24ea785a/global-pull-secret-syncer/0.log" Jul 06 19:51:46.514925 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:46.514898 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6gm25_99098e94-f6ae-4918-b007-e85999dfe259/konnectivity-agent/0.log" Jul 06 19:51:46.637446 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:46.637355 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-237.ec2.internal_62e528d50850ba268ed5e671edfd2eb2/haproxy/0.log" Jul 06 19:51:52.084146 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:52.084117 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-kr5j9_2ab81f98-e81a-4b24-beae-8babf214f1fd/manager/3.log" Jul 06 19:51:52.084998 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:52.084979 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-kr5j9_2ab81f98-e81a-4b24-beae-8babf214f1fd/manager/2.log" Jul 06 19:51:52.121561 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:52.121535 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-wklwt_632d5de0-2e1b-4124-906e-21a096713b69/manager/3.log" Jul 06 19:51:52.129833 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:52.129814 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-wklwt_632d5de0-2e1b-4124-906e-21a096713b69/manager/4.log" Jul 06 19:51:52.167873 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:52.167853 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-vcvtg_ec841e47-d3ca-43bf-a062-361f912c5f27/kuadrant-console-plugin/0.log" Jul 06 19:51:52.244172 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:52.244146 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6tkch_5c49682e-63e9-4178-91ca-a2021cc916ed/registry-server/0.log" Jul 06 19:51:52.398966 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:52.398881 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-7w2nd_995c4fe8-11b1-4e0b-8190-e9c686f0a274/manager/0.log" Jul 06 19:51:52.445706 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:52.445681 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-xh6vt_18559278-46e1-4a51-a067-8a11b7414264/limitador/0.log" Jul 06 19:51:54.493320 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:54.493292 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-s259v_2caf149a-9d45-4412-8c6f-006d28f81ef1/cluster-monitoring-operator/0.log" Jul 06 19:51:54.863276 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:54.863192 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fcdhv_838f35e0-7210-421e-9dc1-3130c82ac0c0/node-exporter/0.log" Jul 06 19:51:54.889517 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:54.889485 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fcdhv_838f35e0-7210-421e-9dc1-3130c82ac0c0/kube-rbac-proxy/0.log" Jul 06 19:51:54.921748 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:54.921715 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fcdhv_838f35e0-7210-421e-9dc1-3130c82ac0c0/init-textfile/0.log" Jul 06 19:51:56.217232 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.217196 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp"] Jul 06 19:51:56.217790 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.217771 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f896f84e-fbab-4513-ac5f-5a613009267c" containerName="test-curl-7551-140020564180688" Jul 06 19:51:56.217887 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.217793 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f896f84e-fbab-4513-ac5f-5a613009267c" containerName="test-curl-7551-140020564180688" Jul 06 19:51:56.217887 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.217882 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f896f84e-fbab-4513-ac5f-5a613009267c" containerName="test-curl-7551-140020564180688" Jul 06 19:51:56.221105 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.221084 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" Jul 06 19:51:56.225387 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.225365 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nqdz2\"/\"default-dockercfg-6xq4g\"" Jul 06 19:51:56.225387 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.225383 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nqdz2\"/\"openshift-service-ca.crt\"" Jul 06 19:51:56.225681 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.225666 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nqdz2\"/\"kube-root-ca.crt\"" Jul 06 19:51:56.232755 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.232733 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp"] Jul 06 19:51:56.240986 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.240961 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/feceebc3-2fe9-46ae-b049-80786d23d018-podres\") pod \"perf-node-gather-daemonset-dw8lp\" (UID: \"feceebc3-2fe9-46ae-b049-80786d23d018\") " pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" Jul 06 19:51:56.241082 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.241024 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/feceebc3-2fe9-46ae-b049-80786d23d018-proc\") pod \"perf-node-gather-daemonset-dw8lp\" (UID: \"feceebc3-2fe9-46ae-b049-80786d23d018\") " pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" Jul 06 19:51:56.241131 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.241116 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/feceebc3-2fe9-46ae-b049-80786d23d018-sys\") pod \"perf-node-gather-daemonset-dw8lp\" (UID: \"feceebc3-2fe9-46ae-b049-80786d23d018\") " pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" Jul 06 19:51:56.241173 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.241139 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/feceebc3-2fe9-46ae-b049-80786d23d018-lib-modules\") pod \"perf-node-gather-daemonset-dw8lp\" (UID: \"feceebc3-2fe9-46ae-b049-80786d23d018\") " pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" Jul 06 19:51:56.241218 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.241207 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnztr\" (UniqueName: \"kubernetes.io/projected/feceebc3-2fe9-46ae-b049-80786d23d018-kube-api-access-lnztr\") pod \"perf-node-gather-daemonset-dw8lp\" (UID: \"feceebc3-2fe9-46ae-b049-80786d23d018\") " pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" Jul 06 19:51:56.342114 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.342070 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/feceebc3-2fe9-46ae-b049-80786d23d018-sys\") pod \"perf-node-gather-daemonset-dw8lp\" (UID: \"feceebc3-2fe9-46ae-b049-80786d23d018\") " pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" Jul 06 19:51:56.342114 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.342111 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/feceebc3-2fe9-46ae-b049-80786d23d018-lib-modules\") pod \"perf-node-gather-daemonset-dw8lp\" (UID: \"feceebc3-2fe9-46ae-b049-80786d23d018\") " pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" Jul 06 19:51:56.342399 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.342209 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/feceebc3-2fe9-46ae-b049-80786d23d018-sys\") pod \"perf-node-gather-daemonset-dw8lp\" (UID: \"feceebc3-2fe9-46ae-b049-80786d23d018\") " pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" Jul 06 19:51:56.342399 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.342225 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/feceebc3-2fe9-46ae-b049-80786d23d018-lib-modules\") pod \"perf-node-gather-daemonset-dw8lp\" (UID: \"feceebc3-2fe9-46ae-b049-80786d23d018\") " pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" Jul 06 19:51:56.342399 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.342327 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnztr\" (UniqueName: \"kubernetes.io/projected/feceebc3-2fe9-46ae-b049-80786d23d018-kube-api-access-lnztr\") pod \"perf-node-gather-daemonset-dw8lp\" (UID: \"feceebc3-2fe9-46ae-b049-80786d23d018\") " pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" Jul 06 19:51:56.342399 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.342381 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/feceebc3-2fe9-46ae-b049-80786d23d018-podres\") pod \"perf-node-gather-daemonset-dw8lp\" (UID: \"feceebc3-2fe9-46ae-b049-80786d23d018\") " pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" Jul 06 19:51:56.342609 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.342504 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/feceebc3-2fe9-46ae-b049-80786d23d018-proc\") pod \"perf-node-gather-daemonset-dw8lp\" (UID: \"feceebc3-2fe9-46ae-b049-80786d23d018\") " pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" Jul 06 19:51:56.342609 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.342552 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/feceebc3-2fe9-46ae-b049-80786d23d018-podres\") pod \"perf-node-gather-daemonset-dw8lp\" (UID: \"feceebc3-2fe9-46ae-b049-80786d23d018\") " pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" Jul 06 19:51:56.342693 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.342613 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/feceebc3-2fe9-46ae-b049-80786d23d018-proc\") pod \"perf-node-gather-daemonset-dw8lp\" (UID: \"feceebc3-2fe9-46ae-b049-80786d23d018\") " pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" Jul 06 19:51:56.358337 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.358306 2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnztr\" (UniqueName: \"kubernetes.io/projected/feceebc3-2fe9-46ae-b049-80786d23d018-kube-api-access-lnztr\") pod \"perf-node-gather-daemonset-dw8lp\" (UID: \"feceebc3-2fe9-46ae-b049-80786d23d018\") " pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" Jul 06 19:51:56.530837 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.530808 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" Jul 06 19:51:56.667976 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.667951 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp"] Jul 06 19:51:56.670802 ip-10-0-141-237 kubenswrapper[2553]: W0706 19:51:56.670777 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfeceebc3_2fe9_46ae_b049_80786d23d018.slice/crio-dd2dcdfe938bb9fd319b01cf66ea269da71a356a704c157eb2fdf1c460cceea3 WatchSource:0}: Error finding container dd2dcdfe938bb9fd319b01cf66ea269da71a356a704c157eb2fdf1c460cceea3: Status 404 returned error can't find the container with id dd2dcdfe938bb9fd319b01cf66ea269da71a356a704c157eb2fdf1c460cceea3 Jul 06 19:51:56.858156 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.858121 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" event={"ID":"feceebc3-2fe9-46ae-b049-80786d23d018","Type":"ContainerStarted","Data":"69f598a0cb55aff423502f3efa5ced3df36fb9c133cb3e66ddb202deee10c9fc"} Jul 06 19:51:56.858300 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.858180 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" event={"ID":"feceebc3-2fe9-46ae-b049-80786d23d018","Type":"ContainerStarted","Data":"dd2dcdfe938bb9fd319b01cf66ea269da71a356a704c157eb2fdf1c460cceea3"} Jul 06 19:51:56.858300 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.858291 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" Jul 06 19:51:56.874697 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:56.874651 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" podStartSLOduration=0.874635613 podStartE2EDuration="874.635613ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:51:56.874324304 +0000 UTC m=+3178.143506647" watchObservedRunningTime="2026-07-06 19:51:56.874635613 +0000 UTC m=+3178.143818008" Jul 06 19:51:57.583925 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:57.583897 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/1.log" Jul 06 19:51:57.592032 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:57.592008 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-8m9wb_1923cd02-24e0-4753-be72-cb22fbd5c05f/console-operator/2.log" Jul 06 19:51:58.817676 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:58.817645 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-b4bv7_13153a33-8136-473a-b96d-7bd6b1e8c5e3/volume-data-source-validator/0.log" Jul 06 19:51:59.542456 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:51:59.542410 2553 scope.go:117] "RemoveContainer" containerID="65737fefcf3e1d0cecb5c03978be00c47999da311ca02ab0002a059f4827bf15" Jul 06 19:52:00.143112 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:00.143084 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mkqz8_41eea561-3913-46d4-beca-714c42bc07cd/dns/0.log" Jul 06 19:52:00.167386 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:00.167361 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mkqz8_41eea561-3913-46d4-beca-714c42bc07cd/kube-rbac-proxy/0.log" Jul 06 19:52:00.278747 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:00.278720 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nb9d8_2f70581d-708c-43ae-871a-a126644cefff/dns-node-resolver/0.log" Jul 06 19:52:00.970887 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:00.970858 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v2xjc_c996c2ca-cb6f-420e-b4d0-5b9720137b7d/node-ca/0.log" Jul 06 19:52:02.045111 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:02.045083 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4j46q6_6ea08144-1c62-42ec-8d15-3da05e0fa17b/istio-proxy/0.log" Jul 06 19:52:02.239144 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:02.239111 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c9545dd54-sjc64_9780aa2b-78ab-489e-be4b-cefc925b9087/kube-auth-proxy/0.log" Jul 06 19:52:02.269970 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:02.269943 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lcj97_6b30dece-56c5-458a-82b9-f799d264cb5a/istio-proxy/0.log" Jul 06 19:52:02.294232 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:02.294203 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lcj97_6b30dece-56c5-458a-82b9-f799d264cb5a/istio-proxy/1.log" Jul 06 19:52:02.469283 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:02.469265 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7456bbb6dc-zfg4t_b78c33c0-d4f4-4a97-8621-90904852db08/router/0.log" Jul 06 19:52:02.873190 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:02.873114 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nqdz2/perf-node-gather-daemonset-dw8lp" Jul 06 19:52:03.442034 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:03.442008 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sc9lq_d11343a5-a7ee-47e9-8875-d6e7ae822930/serve-healthcheck-canary/0.log" Jul 06 19:52:03.963381 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:03.963353 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-tsj58_724d10df-ec14-4dcd-8b9c-5265acf5ed80/insights-operator/0.log" Jul 06 19:52:03.964152 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:03.964136 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-tsj58_724d10df-ec14-4dcd-8b9c-5265acf5ed80/insights-operator/1.log" Jul 06 19:52:04.145914 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:04.145865 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s9tch_b345cada-d7a4-4b8f-af05-29d6b7f91edc/kube-rbac-proxy/0.log" Jul 06 19:52:04.183339 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:04.183310 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s9tch_b345cada-d7a4-4b8f-af05-29d6b7f91edc/exporter/0.log" Jul 06 19:52:04.209346 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:04.209317 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s9tch_b345cada-d7a4-4b8f-af05-29d6b7f91edc/extractor/0.log" Jul 06 19:52:06.160983 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:06.160951 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4j46q6_6ea08144-1c62-42ec-8d15-3da05e0fa17b/istio-proxy/0.log" Jul 06 19:52:06.320363 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:06.320298 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c9545dd54-sjc64_9780aa2b-78ab-489e-be4b-cefc925b9087/kube-auth-proxy/0.log" Jul 06 19:52:06.343818 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:06.343787 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lcj97_6b30dece-56c5-458a-82b9-f799d264cb5a/istio-proxy/0.log" Jul 06 19:52:06.373190 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:06.373155 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lcj97_6b30dece-56c5-458a-82b9-f799d264cb5a/istio-proxy/1.log" Jul 06 19:52:06.506781 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:06.506754 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7456bbb6dc-zfg4t_b78c33c0-d4f4-4a97-8621-90904852db08/router/0.log" Jul 06 19:52:06.911864 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:06.911838 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gwbxn_ad5da93d-e1e5-4854-98fb-7d8f95143b13/manager/3.log" Jul 06 19:52:06.912397 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:06.912378 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gwbxn_ad5da93d-e1e5-4854-98fb-7d8f95143b13/manager/4.log" Jul 06 19:52:06.936200 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:06.936170 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-sq8dz_ba75546a-9e3d-4363-a68d-b194380ff105/manager/3.log" Jul 06 19:52:07.041034 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:07.040988 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-sq8dz_ba75546a-9e3d-4363-a68d-b194380ff105/manager/4.log" Jul 06 19:52:07.099735 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:07.099709 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722755-55kz6_d82ab975-83d9-4c4b-93ee-13c52fced6b3/cleanup/0.log" Jul 06 19:52:07.366747 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:07.366722 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-5psq2_d59bbf4a-f17c-43a9-ab25-d64b4b351819/server/0.log" Jul 06 19:52:07.492399 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:07.492361 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w_c9fa46fe-a84d-449b-a63b-47121b01ba82/manager/0.log" Jul 06 19:52:07.515285 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:07.515261 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58d5dcdfbc-4ht4w_c9fa46fe-a84d-449b-a63b-47121b01ba82/copy-manifests/0.log" Jul 06 19:52:07.727051 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:07.727023 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4j46q6_6ea08144-1c62-42ec-8d15-3da05e0fa17b/istio-proxy/0.log" Jul 06 19:52:07.883624 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:07.883552 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c9545dd54-sjc64_9780aa2b-78ab-489e-be4b-cefc925b9087/kube-auth-proxy/0.log" Jul 06 19:52:07.910757 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:07.910737 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lcj97_6b30dece-56c5-458a-82b9-f799d264cb5a/istio-proxy/0.log" Jul 06 19:52:07.936397 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:07.936372 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-lcj97_6b30dece-56c5-458a-82b9-f799d264cb5a/istio-proxy/1.log" Jul 06 19:52:08.074695 ip-10-0-141-237 kubenswrapper[2553]: I0706 19:52:08.074668 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7456bbb6dc-zfg4t_b78c33c0-d4f4-4a97-8621-90904852db08/router/0.log"