Jul 03 10:03:21.224298 ip-10-0-143-105 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 03 10:03:21.224313 ip-10-0-143-105 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 03 10:03:21.224324 ip-10-0-143-105 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 03 10:03:21.224943 ip-10-0-143-105 systemd[1]: Failed to start Kubernetes Kubelet. Jul 03 10:03:31.286969 ip-10-0-143-105 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 03 10:03:31.286988 ip-10-0-143-105 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot a5e08f9464104c679d788581ef574de2 -- Jul 03 10:05:28.838510 ip-10-0-143-105 systemd[1]: Starting Kubernetes Kubelet... Jul 03 10:05:29.290745 ip-10-0-143-105 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 03 10:05:29.290745 ip-10-0-143-105 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 03 10:05:29.290745 ip-10-0-143-105 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 03 10:05:29.290745 ip-10-0-143-105 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 03 10:05:29.290745 ip-10-0-143-105 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 03 10:05:29.290745 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.254015 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 03 10:05:29.290745 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258751 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 10:05:29.290745 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258766 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 10:05:29.290745 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258770 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 10:05:29.290745 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258773 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 10:05:29.290745 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258776 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 10:05:29.290745 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258779 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 10:05:29.290745 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258782 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 10:05:29.290745 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258785 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 10:05:29.292994 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258788 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 10:05:29.292994 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258793 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 10:05:29.292994 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258797 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 10:05:29.292994 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258800 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 10:05:29.292994 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258802 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 10:05:29.292994 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258805 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 10:05:29.292994 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258808 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 10:05:29.292994 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258810 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 10:05:29.292994 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258813 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 10:05:29.292994 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258816 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 10:05:29.292994 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258818 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 10:05:29.292994 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258821 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 10:05:29.292994 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258823 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 10:05:29.292994 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258826 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 10:05:29.292994 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258829 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 10:05:29.292994 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258837 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 10:05:29.292994 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258841 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 10:05:29.292994 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258843 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 10:05:29.292994 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258846 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258848 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258851 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258854 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258857 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258859 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258863 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258865 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258868 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258871 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258873 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258876 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258878 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258883 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258886 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258888 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258891 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258893 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258897 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258900 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 10:05:29.293571 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258902 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 10:05:29.294100 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258905 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 10:05:29.294100 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258907 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 10:05:29.294100 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258910 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 10:05:29.294100 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258912 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 10:05:29.294100 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258915 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 10:05:29.294100 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258917 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 10:05:29.294100 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258920 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 10:05:29.294100 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258924 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 10:05:29.294100 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258928 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 10:05:29.294100 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258932 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 10:05:29.294100 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258934 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 10:05:29.294100 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258937 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 10:05:29.294100 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258940 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 03 10:05:29.294100 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258944 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 10:05:29.294100 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258946 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 10:05:29.294100 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258951 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 10:05:29.294100 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258954 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 10:05:29.294100 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258957 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 10:05:29.294100 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258960 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.258962 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259731 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259737 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259741 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259744 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259749 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259752 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259756 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259759 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259762 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259765 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259768 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259771 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259774 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259777 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259780 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259783 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259785 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259789 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 10:05:29.294676 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259792 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 10:05:29.295395 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259795 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 10:05:29.295395 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259797 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 10:05:29.295395 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259800 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 10:05:29.295395 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259803 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 10:05:29.295395 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259806 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 10:05:29.295395 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259809 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 10:05:29.295395 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259812 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 10:05:29.295395 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259815 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 10:05:29.295395 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259817 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 10:05:29.295395 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259820 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 10:05:29.295395 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259822 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 10:05:29.295395 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259825 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 10:05:29.295395 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.259827 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 10:05:29.295395 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260556 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 10:05:29.295395 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260564 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 10:05:29.295395 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260567 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 10:05:29.295395 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260569 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 10:05:29.295395 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260572 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 10:05:29.295395 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260575 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260578 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260582 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260584 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260587 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260590 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260592 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260595 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260598 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260600 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260603 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260606 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260608 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260611 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260613 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260616 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260618 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260621 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260624 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260627 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 10:05:29.452609 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260629 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 10:05:29.330587 ip-10-0-143-105 systemd[1]: Started Kubernetes Kubelet. Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260632 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260634 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260639 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260642 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260646 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260649 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260651 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260655 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260658 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260661 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260664 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260668 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260672 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260675 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260678 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260680 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260683 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260686 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260689 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 10:05:29.465980 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260692 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 10:05:29.466814 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260694 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 10:05:29.466814 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260698 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 10:05:29.466814 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260700 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 10:05:29.466814 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260703 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 10:05:29.466814 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260706 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 10:05:29.466814 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260708 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 10:05:29.466814 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260710 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 10:05:29.466814 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260714 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 10:05:29.466814 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260718 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 10:05:29.466814 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260720 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 10:05:29.466814 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260723 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 10:05:29.466814 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260727 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 10:05:29.466814 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260730 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 10:05:29.466814 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260732 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 10:05:29.466814 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260735 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 10:05:29.466814 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260737 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 10:05:29.466814 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260740 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 10:05:29.466814 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260742 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 10:05:29.466814 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260745 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260747 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260750 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260752 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260755 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260758 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260761 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260766 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260768 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260771 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260774 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260777 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260779 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260782 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260784 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260787 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260789 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260792 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260794 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260797 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 10:05:29.467669 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260799 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260802 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260804 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260809 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260812 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260814 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260817 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260819 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260822 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260825 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260827 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260830 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260832 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260835 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.260837 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261299 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261305 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261308 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261312 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261315 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 10:05:29.468457 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261318 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261321 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261323 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261326 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261329 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261331 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261334 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261336 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261339 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261342 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261345 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261347 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261350 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261353 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261355 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261362 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261365 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261367 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261370 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261372 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 10:05:29.469046 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261375 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261377 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261380 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261384 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261387 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261390 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261393 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261396 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261398 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261401 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261403 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261406 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261408 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261412 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261415 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261417 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261420 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261422 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261425 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261429 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 10:05:29.469723 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261431 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261434 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261436 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261439 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261442 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261444 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261446 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261450 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261453 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261455 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261458 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261460 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261463 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261466 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261468 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261471 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261473 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261476 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261478 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261481 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 10:05:29.470320 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261484 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 10:05:29.470933 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261486 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 10:05:29.470933 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261489 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 10:05:29.470933 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261491 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 10:05:29.470933 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261494 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 10:05:29.470933 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261497 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 10:05:29.470933 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261499 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 10:05:29.470933 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261502 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 10:05:29.470933 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261504 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 10:05:29.470933 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261507 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 10:05:29.470933 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261511 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 10:05:29.470933 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261515 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 10:05:29.470933 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261518 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 10:05:29.470933 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261520 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 10:05:29.470933 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261523 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 10:05:29.470933 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261526 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 10:05:29.470933 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261528 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 10:05:29.470933 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261531 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 10:05:29.470933 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261533 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 10:05:29.470933 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261536 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261539 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261542 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261546 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261550 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261552 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261555 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261557 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261560 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261563 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261565 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261567 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261570 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261573 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.261575 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261641 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261648 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261654 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261660 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261667 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261672 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 03 10:05:29.471518 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261678 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261685 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261689 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261694 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261697 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261700 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261704 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261707 2579 flags.go:64] FLAG: --cgroup-root="" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261709 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261712 2579 flags.go:64] FLAG: --client-ca-file="" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261715 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261719 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261724 2579 flags.go:64] FLAG: --cluster-domain="" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261727 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261730 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261733 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261737 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261741 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261744 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261747 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261751 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261753 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261756 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261759 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 03 10:05:29.472159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261762 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261765 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261769 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261772 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261775 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261778 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261781 2579 flags.go:64] FLAG: --enable-server="true" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261784 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261788 2579 flags.go:64] FLAG: --event-burst="100" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261791 2579 flags.go:64] FLAG: --event-qps="50" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261794 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261798 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261801 2579 flags.go:64] FLAG: --eviction-hard="" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261805 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261808 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261811 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261814 2579 flags.go:64] FLAG: --eviction-soft="" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261817 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261819 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261823 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261826 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261830 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261832 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261836 2579 flags.go:64] FLAG: --feature-gates="" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261839 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 03 10:05:29.472862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261842 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261845 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261848 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261851 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261854 2579 flags.go:64] FLAG: --help="false" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261857 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-143-105.ec2.internal" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261861 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261864 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261866 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261870 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261873 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261876 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261878 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261881 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261884 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261887 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261890 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261893 2579 flags.go:64] FLAG: --kube-reserved="" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261897 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261899 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261902 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261905 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261908 2579 flags.go:64] FLAG: --lock-file="" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261910 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 03 10:05:29.473670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261913 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261916 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261922 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261925 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261928 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261931 2579 flags.go:64] FLAG: --logging-format="text" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261934 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261937 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261940 2579 flags.go:64] FLAG: --manifest-url="" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261943 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261947 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261950 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261957 2579 flags.go:64] FLAG: --max-pods="110" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261960 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261963 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261966 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261969 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261972 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261975 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261977 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261985 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261988 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261991 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 03 10:05:29.474332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261994 2579 flags.go:64] FLAG: --pod-cidr="" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.261997 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262002 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262008 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262011 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262014 2579 flags.go:64] FLAG: --port="10250" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262017 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262020 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a8dbe0a58fa27eda" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262023 2579 flags.go:64] FLAG: --qos-reserved="" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262026 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262029 2579 flags.go:64] FLAG: --register-node="true" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262041 2579 flags.go:64] FLAG: --register-with-taints="" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262045 2579 flags.go:64] FLAG: --registry-burst="10" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262048 2579 flags.go:64] FLAG: --registry-qps="5" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262051 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262054 2579 flags.go:64] FLAG: --reserved-memory="" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262058 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262061 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262064 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262067 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262070 2579 flags.go:64] FLAG: --runonce="false" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262072 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262075 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262079 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262081 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 03 10:05:29.474968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262084 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262092 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262095 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262098 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262101 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262104 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262107 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262110 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262113 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262116 2579 flags.go:64] FLAG: --system-cgroups="" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262121 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262126 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262129 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262132 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262136 2579 flags.go:64] FLAG: --tls-min-version="" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262139 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262141 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262144 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262147 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262151 2579 flags.go:64] FLAG: --v="2" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262155 2579 flags.go:64] FLAG: --version="false" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262159 2579 flags.go:64] FLAG: --vmodule="" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262163 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262166 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262296 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 10:05:29.475941 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262301 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 10:05:29.476654 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262304 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 10:05:29.476654 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262307 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 10:05:29.476654 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262310 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 10:05:29.476654 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262313 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 10:05:29.476654 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262316 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 10:05:29.476654 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262319 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 10:05:29.476654 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262321 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 10:05:29.476654 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262324 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 10:05:29.476654 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262326 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 10:05:29.476654 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262329 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 10:05:29.476654 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262332 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 10:05:29.476654 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262334 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 10:05:29.476654 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262337 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 10:05:29.476654 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262340 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 10:05:29.476654 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262343 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 10:05:29.476654 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262345 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 10:05:29.476654 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262350 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 10:05:29.476654 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262353 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 10:05:29.476654 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262355 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262359 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262363 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262366 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262369 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262372 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262374 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262378 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262381 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262384 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262386 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262389 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262392 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262394 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262397 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262400 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262403 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262406 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262409 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262412 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 10:05:29.601939 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262414 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 10:05:29.602577 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262417 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 10:05:29.602577 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262420 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 10:05:29.602577 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262423 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 10:05:29.602577 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262426 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 10:05:29.602577 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262429 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 10:05:29.602577 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262432 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 10:05:29.602577 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262434 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 10:05:29.602577 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262437 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 10:05:29.602577 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262439 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 10:05:29.602577 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262443 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 10:05:29.602577 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262446 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 10:05:29.602577 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262449 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 10:05:29.602577 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262452 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 10:05:29.602577 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262454 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 10:05:29.602577 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262457 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 10:05:29.602577 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262459 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 10:05:29.602577 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262462 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 10:05:29.602577 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262464 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 10:05:29.602577 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262468 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262471 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262473 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262476 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262478 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262481 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262484 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262486 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262489 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262491 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262494 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262497 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262499 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262502 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262504 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262506 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262509 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262512 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262514 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262517 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 10:05:29.603089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262519 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 10:05:29.603675 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262522 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 10:05:29.603675 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262524 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 10:05:29.603675 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262531 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 10:05:29.603675 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262534 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 10:05:29.603675 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262537 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 10:05:29.603675 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262539 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 10:05:29.603675 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262542 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 10:05:29.603675 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262544 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 10:05:29.603675 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262547 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 10:05:29.603675 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262549 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 10:05:29.603675 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262552 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 10:05:29.603675 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262555 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 10:05:29.603675 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262558 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 10:05:29.603675 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262560 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 10:05:29.603675 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262564 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 10:05:29.603675 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262567 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 10:05:29.603675 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262570 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 10:05:29.603675 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262574 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 10:05:29.603675 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.262577 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 10:05:29.604171 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.262587 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 03 10:05:29.604171 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.269281 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 03 10:05:29.604171 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.269296 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 03 10:05:29.604171 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.269312 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 03 10:05:29.604171 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.269318 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 03 10:05:29.604171 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269373 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 10:05:29.604171 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269379 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 10:05:29.604171 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269382 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 10:05:29.604171 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269385 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 10:05:29.604171 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269388 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 10:05:29.604171 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269391 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 10:05:29.604171 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269395 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 10:05:29.604171 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269399 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 10:05:29.604171 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269402 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 10:05:29.604171 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269405 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269408 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269411 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269415 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269417 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269420 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269422 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269425 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269428 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269430 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269433 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269435 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269438 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269442 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269445 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269447 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269450 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269452 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269455 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269458 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 10:05:29.604643 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269461 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 10:05:29.605162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269465 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 10:05:29.605162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269467 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 10:05:29.605162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269470 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 10:05:29.605162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269473 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 10:05:29.605162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269477 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 10:05:29.605162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269480 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 10:05:29.605162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269483 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 10:05:29.605162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269485 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 10:05:29.605162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269488 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 10:05:29.605162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269490 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 10:05:29.605162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269493 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 10:05:29.605162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269495 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 10:05:29.605162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269498 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 10:05:29.605162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269501 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 10:05:29.605162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269503 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 03 10:05:29.605162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269506 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 10:05:29.605162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269509 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 10:05:29.605162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269512 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 10:05:29.605162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269517 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269520 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269523 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269525 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269528 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269531 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269533 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269536 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269538 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269541 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269543 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269546 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269549 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269551 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269554 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269556 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269559 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269562 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269565 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269567 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 10:05:29.605768 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269570 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 10:05:29.606319 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269574 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 10:05:29.606319 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269578 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 10:05:29.606319 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269580 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 10:05:29.606319 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269583 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 10:05:29.606319 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269585 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 10:05:29.606319 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269588 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 10:05:29.606319 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269590 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 10:05:29.606319 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269593 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 10:05:29.606319 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269595 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 10:05:29.606319 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269598 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 10:05:29.606319 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269601 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 10:05:29.606319 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269605 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 10:05:29.606319 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269608 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 10:05:29.606319 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269610 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 10:05:29.606319 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269613 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 10:05:29.606319 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269616 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 10:05:29.606319 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269618 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 10:05:29.606319 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269621 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 10:05:29.606319 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269624 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 10:05:29.606867 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269626 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 10:05:29.606867 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269629 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 10:05:29.606867 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269631 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 10:05:29.606867 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269634 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 10:05:29.606867 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269638 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 10:05:29.606867 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269641 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 10:05:29.606867 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269643 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 10:05:29.606867 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269646 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 10:05:29.606867 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269649 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 10:05:29.606867 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269651 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 10:05:29.606867 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269654 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 10:05:29.606867 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.269659 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 03 10:05:29.606867 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269776 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 10:05:29.606867 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269781 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 10:05:29.606867 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269784 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 10:05:29.607289 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269787 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 10:05:29.607289 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269791 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 10:05:29.607289 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269794 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 10:05:29.607289 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269797 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 10:05:29.607289 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269800 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 10:05:29.607289 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269803 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 10:05:29.607289 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269805 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 10:05:29.607289 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269808 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 10:05:29.607289 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269811 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 10:05:29.607289 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269814 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 10:05:29.607289 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269818 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 10:05:29.607289 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269821 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 10:05:29.607289 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269823 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 10:05:29.607289 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269826 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 03 10:05:29.607289 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269828 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 10:05:29.607289 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269831 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 10:05:29.607289 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269833 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 10:05:29.607289 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269836 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 10:05:29.607289 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269838 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 10:05:29.607839 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269841 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 10:05:29.607839 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269844 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 10:05:29.607839 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269848 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 10:05:29.607839 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269851 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 10:05:29.607839 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269855 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 10:05:29.607839 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269859 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 10:05:29.607839 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269862 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 10:05:29.607839 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269865 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 10:05:29.607839 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269867 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 10:05:29.607839 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269870 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 10:05:29.607839 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269873 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 10:05:29.607839 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269875 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 10:05:29.607839 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269878 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 10:05:29.607839 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269881 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 10:05:29.607839 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269883 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 10:05:29.607839 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269886 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 10:05:29.607839 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269888 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 10:05:29.607839 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269891 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 10:05:29.607839 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269893 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269896 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269898 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269901 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269904 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269907 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269910 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269912 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269915 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269918 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269920 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269923 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269925 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269928 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269930 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269933 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269937 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269939 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269942 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269944 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 10:05:29.608351 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269947 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269950 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269952 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269954 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269957 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269959 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269962 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269964 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269967 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269969 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269972 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269974 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269976 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269979 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269982 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269984 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269987 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269991 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269993 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269996 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 10:05:29.608918 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.269998 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 10:05:29.609467 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.270001 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 10:05:29.609467 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.270003 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 10:05:29.609467 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.270006 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 10:05:29.609467 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.270009 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 10:05:29.609467 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.270011 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 10:05:29.609467 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.270014 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 10:05:29.609467 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.270016 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 10:05:29.609467 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.270020 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 10:05:29.609467 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.270022 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 10:05:29.609467 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.270025 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 10:05:29.609467 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.270027 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 10:05:29.609467 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.270030 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 10:05:29.609467 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.270032 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 10:05:29.609467 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.270034 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 10:05:29.609467 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.270037 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 10:05:29.609467 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.270039 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 10:05:29.609467 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.270042 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 10:05:29.609467 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:29.270044 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 10:05:29.610030 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.270050 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 03 10:05:29.610030 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.270147 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 03 10:05:29.610030 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.272082 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 03 10:05:29.610030 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.273669 2579 server.go:1019] "Starting client certificate rotation" Jul 03 10:05:29.610030 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.273782 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 03 10:05:29.610030 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.273819 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 03 10:05:29.610030 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.278185 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 03 10:05:29.610030 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.279434 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 03 10:05:29.610030 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.294602 2579 log.go:25] "Validated CRI v1 runtime API" Jul 03 10:05:29.610030 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.297973 2579 log.go:25] "Validated CRI v1 image API" Jul 03 10:05:29.610030 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.299189 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 03 10:05:29.610030 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.301538 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8ee5b3cc-22ab-4804-80b9-17e9378048ab:/dev/nvme0n1p4 9f2b9ea1-46d1-4626-bcbe-e77a97ec0ef1:/dev/nvme0n1p3] Jul 03 10:05:29.610427 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.301552 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 03 10:05:29.610532 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.308509 2579 manager.go:216] Machine: {Timestamp:2026-07-03 10:05:29.307076842 +0000 UTC m=+0.345223442 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100225 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b173113f268e4e3fd1ee1070fa0e0 SystemUUID:ec2b1731-13f2-68e4-e3fd-1ee1070fa0e0 BootID:a5e08f94-6410-4c67-9d78-8581ef574de2 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:67:3b:2e:48:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:67:3b:2e:48:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:56:74:8f:dc:94 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 03 10:05:29.610532 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.308646 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 03 10:05:29.610532 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.308803 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 03 10:05:29.610532 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.311706 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 03 10:05:29.610532 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.311739 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-105.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 03 10:05:29.610532 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.312524 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 03 10:05:29.610532 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.312539 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 03 10:05:29.610532 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.312572 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.312906 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.313236 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.314630 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.318021 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.318136 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.318164 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.318176 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.318200 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.318211 2579 kubelet.go:398] "Adding apiserver pod source" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.318237 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.318587 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.318602 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.321029 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.323330 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.323359 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.324403 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.324416 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.324422 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.324428 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.324437 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.324443 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.324450 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.324456 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.324464 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.324469 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.324478 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.324483 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.325501 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.325645 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.329696 2579 server.go:1268] "Started kubelet" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.329779 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.329773 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.329846 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.330290 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-105.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:29.330892 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:29.330893 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.331319 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.332244 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 03 10:05:29.610809 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:29.334339 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.334915 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-482t7" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.337174 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.337183 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:29.337735 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-105.ec2.internal\" not found" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.337901 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.337916 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.337933 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.338011 2579 factory.go:55] Registering systemd factory Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.338026 2579 factory.go:223] Registration of the systemd container factory successfully Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.338055 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.338065 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.338256 2579 factory.go:153] Registering CRI-O factory Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.338270 2579 factory.go:223] Registration of the crio container factory successfully Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.338327 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.338350 2579 factory.go:103] Registering Raw factory Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.338366 2579 manager.go:1196] Started watching for new ooms in manager Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.338648 2579 manager.go:349] Starting recovery of all containers Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.344396 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-482t7" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:29.344514 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:29.345318 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.346118 2579 manager.go:354] Recovery completed Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:29.345009 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-105.ec2.internal.18bebf90623d3520 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-105.ec2.internal,UID:ip-10-0-143-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-105.ec2.internal,},FirstTimestamp:2026-07-03 10:05:29.329669408 +0000 UTC m=+0.367816007,LastTimestamp:2026-07-03 10:05:29.329669408 +0000 UTC m=+0.367816007,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-105.ec2.internal,}" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.354426 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.356562 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasSufficientMemory" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.356586 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.356595 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasSufficientPID" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.357106 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.357107 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.357121 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.357137 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.357769 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasSufficientMemory" Jul 03 10:05:29.612017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.357790 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.357801 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasSufficientPID" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.358317 2579 policy_none.go:49] "None policy: Start" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.358342 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.358355 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.359298 2579 policy_none.go:47] "Start" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.407760 2579 manager.go:337] "Starting Device Plugin manager" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:29.407928 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.407939 2579 server.go:87] "Starting device plugin registration server" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.408150 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.408160 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.408275 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.408358 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.408367 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:29.408811 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:29.408848 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-105.ec2.internal\" not found" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.508629 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.509957 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasSufficientMemory" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.509994 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.510011 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasSufficientPID" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.510037 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-105.ec2.internal" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.519093 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-105.ec2.internal" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:29.519113 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-105.ec2.internal\": node \"ip-10-0-143-105.ec2.internal\" not found" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.522996 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.524388 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.524425 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.524445 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:29.524475 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.527586 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 10:05:29.615062 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:29.537869 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-105.ec2.internal\" not found" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.626388 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-105.ec2.internal"] Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.626467 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.628105 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasSufficientMemory" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.628135 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.628150 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasSufficientPID" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.629528 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.629658 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.629687 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.630548 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasSufficientMemory" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.630562 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasSufficientMemory" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.630579 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.630582 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.630589 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasSufficientPID" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.630592 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasSufficientPID" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.632136 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.ec2.internal" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.632158 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.635363 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasSufficientMemory" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.635397 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.635414 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeHasSufficientPID" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:29.638960 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-105.ec2.internal\" not found" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:29.667185 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-105.ec2.internal\" not found" node="ip-10-0-143-105.ec2.internal" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:29.671658 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-105.ec2.internal\" not found" node="ip-10-0-143-105.ec2.internal" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:29.739045 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-105.ec2.internal\" not found" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.739091 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51aae6dcfb54935c8a4ffb1745f8057f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal\" (UID: \"51aae6dcfb54935c8a4ffb1745f8057f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.739127 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51aae6dcfb54935c8a4ffb1745f8057f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal\" (UID: \"51aae6dcfb54935c8a4ffb1745f8057f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal" Jul 03 10:05:29.805630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.739146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b144ac3fd0711db4037382f38ebdd87-config\") pod \"kube-apiserver-proxy-ip-10-0-143-105.ec2.internal\" (UID: \"6b144ac3fd0711db4037382f38ebdd87\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.ec2.internal" Jul 03 10:05:29.978885 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:29.839853 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-105.ec2.internal\" not found" Jul 03 10:05:29.978885 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.839917 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51aae6dcfb54935c8a4ffb1745f8057f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal\" (UID: \"51aae6dcfb54935c8a4ffb1745f8057f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal" Jul 03 10:05:29.978885 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.839940 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51aae6dcfb54935c8a4ffb1745f8057f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal\" (UID: \"51aae6dcfb54935c8a4ffb1745f8057f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal" Jul 03 10:05:29.978885 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.839960 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b144ac3fd0711db4037382f38ebdd87-config\") pod \"kube-apiserver-proxy-ip-10-0-143-105.ec2.internal\" (UID: \"6b144ac3fd0711db4037382f38ebdd87\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.ec2.internal" Jul 03 10:05:29.978885 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.840013 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51aae6dcfb54935c8a4ffb1745f8057f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal\" (UID: \"51aae6dcfb54935c8a4ffb1745f8057f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal" Jul 03 10:05:29.978885 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.840097 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51aae6dcfb54935c8a4ffb1745f8057f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal\" (UID: \"51aae6dcfb54935c8a4ffb1745f8057f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal" Jul 03 10:05:29.978885 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.840142 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b144ac3fd0711db4037382f38ebdd87-config\") pod \"kube-apiserver-proxy-ip-10-0-143-105.ec2.internal\" (UID: \"6b144ac3fd0711db4037382f38ebdd87\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.ec2.internal" Jul 03 10:05:29.978885 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:29.945122 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-105.ec2.internal\" not found" Jul 03 10:05:29.978885 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.971170 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal" Jul 03 10:05:29.978885 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:29.975167 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.ec2.internal" Jul 03 10:05:30.120231 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:30.045953 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-105.ec2.internal\" not found" Jul 03 10:05:30.302037 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:30.146935 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-105.ec2.internal\" not found" Jul 03 10:05:30.302037 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:30.247560 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-105.ec2.internal\" not found" Jul 03 10:05:30.302037 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:30.272887 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 03 10:05:30.302037 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:30.273045 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 03 10:05:30.506730 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:30.347762 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-02 10:00:29 +0000 UTC" deadline="2028-02-15 04:24:46.864548016 +0000 UTC" Jul 03 10:05:30.506730 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:30.347786 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14202h19m16.516765321s" Jul 03 10:05:30.506730 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:30.347794 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-105.ec2.internal\" not found" Jul 03 10:05:30.506730 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:30.448407 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-105.ec2.internal\" not found" Jul 03 10:05:30.506730 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:30.475092 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 10:05:30.506730 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:30.494046 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 10:05:30.679006 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:30.538357 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal" Jul 03 10:05:30.679006 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:30.554329 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 03 10:05:30.679006 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:30.555471 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.ec2.internal" Jul 03 10:05:30.679006 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:30.566089 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 03 10:05:30.679006 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:30.667781 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b144ac3fd0711db4037382f38ebdd87.slice/crio-ec8d1f12c70d2956b0684c97403899c0d3431399aed5a59ee9fd4df074bb4eb5 WatchSource:0}: Error finding container ec8d1f12c70d2956b0684c97403899c0d3431399aed5a59ee9fd4df074bb4eb5: Status 404 returned error can't find the container with id ec8d1f12c70d2956b0684c97403899c0d3431399aed5a59ee9fd4df074bb4eb5 Jul 03 10:05:30.679006 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:30.668286 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51aae6dcfb54935c8a4ffb1745f8057f.slice/crio-4c40b1b77e1151d1286ab4ecceaf85a8a11c1ca5f2775296ee0a26d43ceb5b1f WatchSource:0}: Error finding container 4c40b1b77e1151d1286ab4ecceaf85a8a11c1ca5f2775296ee0a26d43ceb5b1f: Status 404 returned error can't find the container with id 4c40b1b77e1151d1286ab4ecceaf85a8a11c1ca5f2775296ee0a26d43ceb5b1f Jul 03 10:05:30.679006 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:30.674028 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 03 10:05:30.878328 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:30.858744 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.319246 2579 apiserver.go:52] "Watching apiserver" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.327845 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.328381 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-105.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg","openshift-cluster-node-tuning-operator/tuned-ktznn","openshift-dns/node-resolver-4hpsh","openshift-multus/multus-additional-cni-plugins-dbhqx","openshift-multus/network-metrics-daemon-lwkn9","openshift-network-diagnostics/network-check-target-pzhjd","kube-system/global-pull-secret-syncer-pzm5l","kube-system/konnectivity-agent-qskxt","openshift-image-registry/node-ca-vbnml","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal","openshift-multus/multus-fbl2t","openshift-network-operator/iptables-alerter-8rwtr","openshift-ovn-kubernetes/ovnkube-node-knz5k"] Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.330246 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vbnml" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.331518 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8rwtr" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.332941 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.333065 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.333132 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.333219 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tmv84\"" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.334016 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.334066 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.334144 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.334310 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.334332 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rt49l\"" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.334856 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4hpsh" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.336691 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.337488 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.338413 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.338540 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.338784 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.338831 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9fbrg\"" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.338992 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lrljg\"" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.339045 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.339321 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.339334 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.339354 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.339362 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 03 10:05:31.372107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.339441 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4spk7\"" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:31.339438 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwkn9" podUID="de5822f2-b52a-4283-938f-0395efeae942" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.339706 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.339725 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.341294 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qskxt" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.343187 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.343501 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.343647 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2hxjr\"" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.343769 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.346187 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.346221 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h8jtr\"" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.346379 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.346817 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.349323 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.349383 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.349589 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.349641 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w72hs\"" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.349908 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.349945 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.350151 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.350292 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.350510 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.351154 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.351615 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:31.351677 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzm5l" podUID="a6273a00-72c1-456e-b47d-9a24d012f8c0" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.352861 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.353815 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:31.374095 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:31.354038 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzhjd" podUID="abbde693-f1f4-469f-bc67-c793cb2007de" Jul 03 10:05:31.375862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.354088 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-b84kz\"" Jul 03 10:05:31.375862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.354773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21ff74d6-895a-4a92-9da8-b13c8db95a92-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.375862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.354814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d-iptables-alerter-script\") pod \"iptables-alerter-8rwtr\" (UID: \"ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d\") " pod="openshift-network-operator/iptables-alerter-8rwtr" Jul 03 10:05:31.375862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.354852 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21ff74d6-895a-4a92-9da8-b13c8db95a92-os-release\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.375862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.354934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs\") pod \"network-metrics-daemon-lwkn9\" (UID: \"de5822f2-b52a-4283-938f-0395efeae942\") " pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:31.375862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.354975 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-sysctl-conf\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.375862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355012 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-run\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.375862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355135 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-var-lib-kubelet\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.375862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355172 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwczl\" (UniqueName: \"kubernetes.io/projected/85cda35a-33a5-4f89-96f4-a2b3f8b585f7-kube-api-access-zwczl\") pod \"node-ca-vbnml\" (UID: \"85cda35a-33a5-4f89-96f4-a2b3f8b585f7\") " pod="openshift-image-registry/node-ca-vbnml" Jul 03 10:05:31.375862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355206 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21ff74d6-895a-4a92-9da8-b13c8db95a92-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.375862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9cfh\" (UniqueName: \"kubernetes.io/projected/de5822f2-b52a-4283-938f-0395efeae942-kube-api-access-t9cfh\") pod \"network-metrics-daemon-lwkn9\" (UID: \"de5822f2-b52a-4283-938f-0395efeae942\") " pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:31.375862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-modprobe-d\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.375862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355320 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-lib-modules\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.375862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355352 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m95h2\" (UniqueName: \"kubernetes.io/projected/21ff74d6-895a-4a92-9da8-b13c8db95a92-kube-api-access-m95h2\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.376632 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-tuned\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.376632 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355412 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8254340-be1e-449d-8022-1d4da25997ff-tmp-dir\") pod \"node-resolver-4hpsh\" (UID: \"f8254340-be1e-449d-8022-1d4da25997ff\") " pod="openshift-dns/node-resolver-4hpsh" Jul 03 10:05:31.376632 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355449 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21ff74d6-895a-4a92-9da8-b13c8db95a92-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.376632 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355486 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85cda35a-33a5-4f89-96f4-a2b3f8b585f7-serviceca\") pod \"node-ca-vbnml\" (UID: \"85cda35a-33a5-4f89-96f4-a2b3f8b585f7\") " pod="openshift-image-registry/node-ca-vbnml" Jul 03 10:05:31.376632 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6e7cb03-364e-4480-84c8-e163592cfa64-agent-certs\") pod \"konnectivity-agent-qskxt\" (UID: \"b6e7cb03-364e-4480-84c8-e163592cfa64\") " pod="kube-system/konnectivity-agent-qskxt" Jul 03 10:05:31.376632 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355584 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d-host-slash\") pod \"iptables-alerter-8rwtr\" (UID: \"ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d\") " pod="openshift-network-operator/iptables-alerter-8rwtr" Jul 03 10:05:31.376632 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355630 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52gxm\" (UniqueName: \"kubernetes.io/projected/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-kube-api-access-52gxm\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.376632 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355793 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-sysconfig\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.376632 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355822 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-sysctl-d\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.376632 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355852 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-systemd\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.376632 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355881 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjclg\" (UniqueName: \"kubernetes.io/projected/f8254340-be1e-449d-8022-1d4da25997ff-kube-api-access-xjclg\") pod \"node-resolver-4hpsh\" (UID: \"f8254340-be1e-449d-8022-1d4da25997ff\") " pod="openshift-dns/node-resolver-4hpsh" Jul 03 10:05:31.376632 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355914 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21ff74d6-895a-4a92-9da8-b13c8db95a92-system-cni-dir\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.376632 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355940 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21ff74d6-895a-4a92-9da8-b13c8db95a92-cnibin\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.376632 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.355970 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6e7cb03-364e-4480-84c8-e163592cfa64-konnectivity-ca\") pod \"konnectivity-agent-qskxt\" (UID: \"b6e7cb03-364e-4480-84c8-e163592cfa64\") " pod="kube-system/konnectivity-agent-qskxt" Jul 03 10:05:31.377761 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.356000 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8vbx\" (UniqueName: \"kubernetes.io/projected/ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d-kube-api-access-f8vbx\") pod \"iptables-alerter-8rwtr\" (UID: \"ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d\") " pod="openshift-network-operator/iptables-alerter-8rwtr" Jul 03 10:05:31.377761 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.356031 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-kubernetes\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.377761 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.356078 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-host\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.377761 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.356141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-tmp\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.377761 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.356180 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85cda35a-33a5-4f89-96f4-a2b3f8b585f7-host\") pod \"node-ca-vbnml\" (UID: \"85cda35a-33a5-4f89-96f4-a2b3f8b585f7\") " pod="openshift-image-registry/node-ca-vbnml" Jul 03 10:05:31.377761 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.356256 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21ff74d6-895a-4a92-9da8-b13c8db95a92-cni-binary-copy\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.377761 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.356287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-sys\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.377761 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.356345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8254340-be1e-449d-8022-1d4da25997ff-hosts-file\") pod \"node-resolver-4hpsh\" (UID: \"f8254340-be1e-449d-8022-1d4da25997ff\") " pod="openshift-dns/node-resolver-4hpsh" Jul 03 10:05:31.377761 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.371106 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cprmz" Jul 03 10:05:31.380365 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.379643 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cprmz" Jul 03 10:05:31.441460 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.441434 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 03 10:05:31.457070 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457035 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21ff74d6-895a-4a92-9da8-b13c8db95a92-cni-binary-copy\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.457195 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457075 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-sys\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.457195 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457118 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-var-lib-openvswitch\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.457195 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf44c15c-90ee-4d0e-afab-882c6620b5c5-socket-dir\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.457195 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-host-run-netns\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.457438 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21ff74d6-895a-4a92-9da8-b13c8db95a92-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.457438 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d-iptables-alerter-script\") pod \"iptables-alerter-8rwtr\" (UID: \"ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d\") " pod="openshift-network-operator/iptables-alerter-8rwtr" Jul 03 10:05:31.457438 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457290 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-run-systemd\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.457438 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-sysctl-conf\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.457438 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457342 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-run\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.457438 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457374 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-var-lib-kubelet\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.457438 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret\") pod \"global-pull-secret-syncer-pzm5l\" (UID: \"a6273a00-72c1-456e-b47d-9a24d012f8c0\") " pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:31.457689 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457430 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-run-openvswitch\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.457689 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457483 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-system-cni-dir\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.457689 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-multus-cni-dir\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.457689 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwczl\" (UniqueName: \"kubernetes.io/projected/85cda35a-33a5-4f89-96f4-a2b3f8b585f7-kube-api-access-zwczl\") pod \"node-ca-vbnml\" (UID: \"85cda35a-33a5-4f89-96f4-a2b3f8b585f7\") " pod="openshift-image-registry/node-ca-vbnml" Jul 03 10:05:31.457689 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-modprobe-d\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.457689 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbd6f\" (UniqueName: \"kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f\") pod \"network-check-target-pzhjd\" (UID: \"abbde693-f1f4-469f-bc67-c793cb2007de\") " pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:31.457689 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-cnibin\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.457689 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457645 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.457689 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457673 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a8b64aea-2cb1-4919-ac4f-e002681cf606-ovnkube-config\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.458018 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457699 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf44c15c-90ee-4d0e-afab-882c6620b5c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.458018 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457723 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf44c15c-90ee-4d0e-afab-882c6620b5c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.458018 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457769 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-host-run-k8s-cni-cncf-io\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.458018 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85cda35a-33a5-4f89-96f4-a2b3f8b585f7-serviceca\") pod \"node-ca-vbnml\" (UID: \"85cda35a-33a5-4f89-96f4-a2b3f8b585f7\") " pod="openshift-image-registry/node-ca-vbnml" Jul 03 10:05:31.458018 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457837 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6e7cb03-364e-4480-84c8-e163592cfa64-agent-certs\") pod \"konnectivity-agent-qskxt\" (UID: \"b6e7cb03-364e-4480-84c8-e163592cfa64\") " pod="kube-system/konnectivity-agent-qskxt" Jul 03 10:05:31.458018 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457863 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxh6v\" (UniqueName: \"kubernetes.io/projected/bf44c15c-90ee-4d0e-afab-882c6620b5c5-kube-api-access-bxh6v\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.458018 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457887 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94799f7a-6383-48ca-8ba2-6d5d0eee3148-cni-binary-copy\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.458018 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457914 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-multus-conf-dir\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.458018 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.457940 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-host-run-multus-certs\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.458508 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.458026 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-sys\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.458508 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.458466 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-run\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.458686 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.458657 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/21ff74d6-895a-4a92-9da8-b13c8db95a92-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.458796 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.458778 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-var-lib-kubelet\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.459024 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.458999 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d-iptables-alerter-script\") pod \"iptables-alerter-8rwtr\" (UID: \"ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d\") " pod="openshift-network-operator/iptables-alerter-8rwtr" Jul 03 10:05:31.459196 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459176 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-sysctl-conf\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.459265 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459191 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21ff74d6-895a-4a92-9da8-b13c8db95a92-cni-binary-copy\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.459265 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459196 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-modprobe-d\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.459265 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjclg\" (UniqueName: \"kubernetes.io/projected/f8254340-be1e-449d-8022-1d4da25997ff-kube-api-access-xjclg\") pod \"node-resolver-4hpsh\" (UID: \"f8254340-be1e-449d-8022-1d4da25997ff\") " pod="openshift-dns/node-resolver-4hpsh" Jul 03 10:05:31.459441 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459320 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-cni-netd\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.459441 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459350 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a8b64aea-2cb1-4919-ac4f-e002681cf606-ovnkube-script-lib\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.459441 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459392 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v84gz\" (UniqueName: \"kubernetes.io/projected/a8b64aea-2cb1-4919-ac4f-e002681cf606-kube-api-access-v84gz\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.459441 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459419 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-os-release\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.459647 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21ff74d6-895a-4a92-9da8-b13c8db95a92-system-cni-dir\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.459647 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6e7cb03-364e-4480-84c8-e163592cfa64-konnectivity-ca\") pod \"konnectivity-agent-qskxt\" (UID: \"b6e7cb03-364e-4480-84c8-e163592cfa64\") " pod="kube-system/konnectivity-agent-qskxt" Jul 03 10:05:31.459647 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459551 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8vbx\" (UniqueName: \"kubernetes.io/projected/ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d-kube-api-access-f8vbx\") pod \"iptables-alerter-8rwtr\" (UID: \"ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d\") " pod="openshift-network-operator/iptables-alerter-8rwtr" Jul 03 10:05:31.459647 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459576 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-kubernetes\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.459647 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-host\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.459647 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459598 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21ff74d6-895a-4a92-9da8-b13c8db95a92-system-cni-dir\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.459647 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459623 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85cda35a-33a5-4f89-96f4-a2b3f8b585f7-serviceca\") pod \"node-ca-vbnml\" (UID: \"85cda35a-33a5-4f89-96f4-a2b3f8b585f7\") " pod="openshift-image-registry/node-ca-vbnml" Jul 03 10:05:31.459647 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-tmp\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.460061 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-run-ovn-kubernetes\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.460061 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459722 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-host\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.460061 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459813 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-kubernetes\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.460061 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459883 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 03 10:05:31.460061 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459933 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85cda35a-33a5-4f89-96f4-a2b3f8b585f7-host\") pod \"node-ca-vbnml\" (UID: \"85cda35a-33a5-4f89-96f4-a2b3f8b585f7\") " pod="openshift-image-registry/node-ca-vbnml" Jul 03 10:05:31.460061 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.459968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8254340-be1e-449d-8022-1d4da25997ff-hosts-file\") pod \"node-resolver-4hpsh\" (UID: \"f8254340-be1e-449d-8022-1d4da25997ff\") " pod="openshift-dns/node-resolver-4hpsh" Jul 03 10:05:31.460061 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460018 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6273a00-72c1-456e-b47d-9a24d012f8c0-dbus\") pod \"global-pull-secret-syncer-pzm5l\" (UID: \"a6273a00-72c1-456e-b47d-9a24d012f8c0\") " pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:31.460451 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460074 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-multus-socket-dir-parent\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.460451 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460129 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-host-var-lib-cni-multus\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.460451 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-host-var-lib-kubelet\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.460451 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460185 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-etc-kubernetes\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.460451 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-hostroot\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.460451 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21ff74d6-895a-4a92-9da8-b13c8db95a92-os-release\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.460451 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460344 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs\") pod \"network-metrics-daemon-lwkn9\" (UID: \"de5822f2-b52a-4283-938f-0395efeae942\") " pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:31.460451 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460373 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-run-ovn\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.460451 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460401 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-log-socket\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.460451 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460426 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf44c15c-90ee-4d0e-afab-882c6620b5c5-sys-fs\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.460451 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460451 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94799f7a-6383-48ca-8ba2-6d5d0eee3148-multus-daemon-config\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.460968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460479 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21ff74d6-895a-4a92-9da8-b13c8db95a92-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.460968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460483 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21ff74d6-895a-4a92-9da8-b13c8db95a92-os-release\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.460968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9cfh\" (UniqueName: \"kubernetes.io/projected/de5822f2-b52a-4283-938f-0395efeae942-kube-api-access-t9cfh\") pod \"network-metrics-daemon-lwkn9\" (UID: \"de5822f2-b52a-4283-938f-0395efeae942\") " pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:31.460968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460711 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-lib-modules\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.460968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-run-netns\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.460968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-etc-openvswitch\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.460968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460800 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a8b64aea-2cb1-4919-ac4f-e002681cf606-ovn-node-metrics-cert\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.460968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4jdc\" (UniqueName: \"kubernetes.io/projected/94799f7a-6383-48ca-8ba2-6d5d0eee3148-kube-api-access-m4jdc\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.460968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460859 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m95h2\" (UniqueName: \"kubernetes.io/projected/21ff74d6-895a-4a92-9da8-b13c8db95a92-kube-api-access-m95h2\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.460968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460904 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-tuned\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.460968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460907 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f8254340-be1e-449d-8022-1d4da25997ff-hosts-file\") pod \"node-resolver-4hpsh\" (UID: \"f8254340-be1e-449d-8022-1d4da25997ff\") " pod="openshift-dns/node-resolver-4hpsh" Jul 03 10:05:31.460968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.460931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8254340-be1e-449d-8022-1d4da25997ff-tmp-dir\") pod \"node-resolver-4hpsh\" (UID: \"f8254340-be1e-449d-8022-1d4da25997ff\") " pod="openshift-dns/node-resolver-4hpsh" Jul 03 10:05:31.461709 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461034 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85cda35a-33a5-4f89-96f4-a2b3f8b585f7-host\") pod \"node-ca-vbnml\" (UID: \"85cda35a-33a5-4f89-96f4-a2b3f8b585f7\") " pod="openshift-image-registry/node-ca-vbnml" Jul 03 10:05:31.461709 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461082 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-kubelet\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.461709 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461122 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-systemd-units\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.461709 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461131 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21ff74d6-895a-4a92-9da8-b13c8db95a92-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.461709 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461139 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-lib-modules\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.461709 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461147 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf44c15c-90ee-4d0e-afab-882c6620b5c5-registration-dir\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.461709 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461182 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6e7cb03-364e-4480-84c8-e163592cfa64-konnectivity-ca\") pod \"konnectivity-agent-qskxt\" (UID: \"b6e7cb03-364e-4480-84c8-e163592cfa64\") " pod="kube-system/konnectivity-agent-qskxt" Jul 03 10:05:31.461709 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21ff74d6-895a-4a92-9da8-b13c8db95a92-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.461709 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d-host-slash\") pod \"iptables-alerter-8rwtr\" (UID: \"ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d\") " pod="openshift-network-operator/iptables-alerter-8rwtr" Jul 03 10:05:31.461709 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461340 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52gxm\" (UniqueName: \"kubernetes.io/projected/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-kube-api-access-52gxm\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.461709 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461370 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a8b64aea-2cb1-4919-ac4f-e002681cf606-env-overrides\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.461709 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf44c15c-90ee-4d0e-afab-882c6620b5c5-device-dir\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.461709 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461454 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8254340-be1e-449d-8022-1d4da25997ff-tmp-dir\") pod \"node-resolver-4hpsh\" (UID: \"f8254340-be1e-449d-8022-1d4da25997ff\") " pod="openshift-dns/node-resolver-4hpsh" Jul 03 10:05:31.461709 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461463 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-sysconfig\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.462146 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-sysctl-d\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.462146 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-systemd\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.462146 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:31.461563 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 10:05:31.462146 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461563 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6273a00-72c1-456e-b47d-9a24d012f8c0-kubelet-config\") pod \"global-pull-secret-syncer-pzm5l\" (UID: \"a6273a00-72c1-456e-b47d-9a24d012f8c0\") " pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:31.462146 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461611 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-sysconfig\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.462146 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461693 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-slash\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.462146 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461706 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-systemd\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.462146 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:31.461720 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs podName:de5822f2-b52a-4283-938f-0395efeae942 nodeName:}" failed. No retries permitted until 2026-07-03 10:05:31.961698438 +0000 UTC m=+2.999845031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs") pod "network-metrics-daemon-lwkn9" (UID: "de5822f2-b52a-4283-938f-0395efeae942") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 10:05:31.462146 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461763 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d-host-slash\") pod \"iptables-alerter-8rwtr\" (UID: \"ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d\") " pod="openshift-network-operator/iptables-alerter-8rwtr" Jul 03 10:05:31.462146 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461823 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-sysctl-d\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.462146 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461839 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/21ff74d6-895a-4a92-9da8-b13c8db95a92-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.462146 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-cni-bin\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.462146 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461895 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-host-var-lib-cni-bin\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.462146 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461930 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21ff74d6-895a-4a92-9da8-b13c8db95a92-cnibin\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.462822 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.461958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-node-log\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.462822 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.462055 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21ff74d6-895a-4a92-9da8-b13c8db95a92-cnibin\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.467075 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.466186 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-tmp\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.467075 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.466857 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6e7cb03-364e-4480-84c8-e163592cfa64-agent-certs\") pod \"konnectivity-agent-qskxt\" (UID: \"b6e7cb03-364e-4480-84c8-e163592cfa64\") " pod="kube-system/konnectivity-agent-qskxt" Jul 03 10:05:31.468144 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.468121 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-etc-tuned\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.470463 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.470440 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8vbx\" (UniqueName: \"kubernetes.io/projected/ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d-kube-api-access-f8vbx\") pod \"iptables-alerter-8rwtr\" (UID: \"ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d\") " pod="openshift-network-operator/iptables-alerter-8rwtr" Jul 03 10:05:31.473031 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.472974 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwczl\" (UniqueName: \"kubernetes.io/projected/85cda35a-33a5-4f89-96f4-a2b3f8b585f7-kube-api-access-zwczl\") pod \"node-ca-vbnml\" (UID: \"85cda35a-33a5-4f89-96f4-a2b3f8b585f7\") " pod="openshift-image-registry/node-ca-vbnml" Jul 03 10:05:31.473775 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.473747 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjclg\" (UniqueName: \"kubernetes.io/projected/f8254340-be1e-449d-8022-1d4da25997ff-kube-api-access-xjclg\") pod \"node-resolver-4hpsh\" (UID: \"f8254340-be1e-449d-8022-1d4da25997ff\") " pod="openshift-dns/node-resolver-4hpsh" Jul 03 10:05:31.473997 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.473973 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m95h2\" (UniqueName: \"kubernetes.io/projected/21ff74d6-895a-4a92-9da8-b13c8db95a92-kube-api-access-m95h2\") pod \"multus-additional-cni-plugins-dbhqx\" (UID: \"21ff74d6-895a-4a92-9da8-b13c8db95a92\") " pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.474615 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.474592 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52gxm\" (UniqueName: \"kubernetes.io/projected/3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf-kube-api-access-52gxm\") pod \"tuned-ktznn\" (UID: \"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf\") " pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.475644 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.475618 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9cfh\" (UniqueName: \"kubernetes.io/projected/de5822f2-b52a-4283-938f-0395efeae942-kube-api-access-t9cfh\") pod \"network-metrics-daemon-lwkn9\" (UID: \"de5822f2-b52a-4283-938f-0395efeae942\") " pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:31.530569 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.530505 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.ec2.internal" event={"ID":"6b144ac3fd0711db4037382f38ebdd87","Type":"ContainerStarted","Data":"ec8d1f12c70d2956b0684c97403899c0d3431399aed5a59ee9fd4df074bb4eb5"} Jul 03 10:05:31.532142 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.532113 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal" event={"ID":"51aae6dcfb54935c8a4ffb1745f8057f","Type":"ContainerStarted","Data":"4c40b1b77e1151d1286ab4ecceaf85a8a11c1ca5f2775296ee0a26d43ceb5b1f"} Jul 03 10:05:31.562992 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.562955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-etc-kubernetes\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.563177 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-hostroot\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.563177 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563075 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-run-ovn\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.563177 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563108 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-etc-kubernetes\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.563177 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-log-socket\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.563177 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563079 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-hostroot\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.563177 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf44c15c-90ee-4d0e-afab-882c6620b5c5-sys-fs\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.563177 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563156 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-log-socket\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.563177 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563157 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-run-ovn\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.563654 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563198 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bf44c15c-90ee-4d0e-afab-882c6620b5c5-sys-fs\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.563654 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94799f7a-6383-48ca-8ba2-6d5d0eee3148-multus-daemon-config\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.563654 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-run-netns\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.563654 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-etc-openvswitch\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.563654 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a8b64aea-2cb1-4919-ac4f-e002681cf606-ovn-node-metrics-cert\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.563654 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563386 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-etc-openvswitch\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.563654 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4jdc\" (UniqueName: \"kubernetes.io/projected/94799f7a-6383-48ca-8ba2-6d5d0eee3148-kube-api-access-m4jdc\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.563654 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563428 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-kubelet\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.563654 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563445 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-systemd-units\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.563654 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563462 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf44c15c-90ee-4d0e-afab-882c6620b5c5-registration-dir\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.563654 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563480 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a8b64aea-2cb1-4919-ac4f-e002681cf606-env-overrides\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.563654 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf44c15c-90ee-4d0e-afab-882c6620b5c5-device-dir\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.563654 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6273a00-72c1-456e-b47d-9a24d012f8c0-kubelet-config\") pod \"global-pull-secret-syncer-pzm5l\" (UID: \"a6273a00-72c1-456e-b47d-9a24d012f8c0\") " pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:31.563654 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-slash\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.564444 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-cni-bin\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.564444 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-host-var-lib-cni-bin\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.564444 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-node-log\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.564444 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-var-lib-openvswitch\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.564444 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563619 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf44c15c-90ee-4d0e-afab-882c6620b5c5-socket-dir\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.564444 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-host-run-netns\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.564444 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-run-systemd\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.564444 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret\") pod \"global-pull-secret-syncer-pzm5l\" (UID: \"a6273a00-72c1-456e-b47d-9a24d012f8c0\") " pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:31.564444 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-run-openvswitch\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.564444 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-system-cni-dir\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.564444 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563752 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-multus-cni-dir\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.564444 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563795 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbd6f\" (UniqueName: \"kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f\") pod \"network-check-target-pzhjd\" (UID: \"abbde693-f1f4-469f-bc67-c793cb2007de\") " pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:31.564444 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563820 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-cnibin\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.564444 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563846 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.565136 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563876 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a8b64aea-2cb1-4919-ac4f-e002681cf606-ovnkube-config\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.565136 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563897 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94799f7a-6383-48ca-8ba2-6d5d0eee3148-multus-daemon-config\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.565136 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563961 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf44c15c-90ee-4d0e-afab-882c6620b5c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.565136 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564008 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-run-netns\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.565136 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.563901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bf44c15c-90ee-4d0e-afab-882c6620b5c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.565136 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564057 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf44c15c-90ee-4d0e-afab-882c6620b5c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.565136 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564087 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-host-run-k8s-cni-cncf-io\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.565136 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564120 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxh6v\" (UniqueName: \"kubernetes.io/projected/bf44c15c-90ee-4d0e-afab-882c6620b5c5-kube-api-access-bxh6v\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.565136 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94799f7a-6383-48ca-8ba2-6d5d0eee3148-cni-binary-copy\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.565136 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-multus-conf-dir\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.565136 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564254 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-var-lib-openvswitch\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.565136 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564272 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-run-systemd\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.565136 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564258 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-cnibin\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.565136 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564282 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-cni-bin\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.565136 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564289 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-host-run-netns\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.565952 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564342 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-node-log\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.565952 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564374 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-slash\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.565952 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564406 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a6273a00-72c1-456e-b47d-9a24d012f8c0-kubelet-config\") pod \"global-pull-secret-syncer-pzm5l\" (UID: \"a6273a00-72c1-456e-b47d-9a24d012f8c0\") " pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:31.565952 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564420 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-run-openvswitch\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.565952 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564454 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-multus-cni-dir\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.565952 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564502 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-systemd-units\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.565952 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:31.564502 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 10:05:31.565952 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564536 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.565952 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564547 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-host-run-k8s-cni-cncf-io\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.565952 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:31.564572 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret podName:a6273a00-72c1-456e-b47d-9a24d012f8c0 nodeName:}" failed. No retries permitted until 2026-07-03 10:05:32.064553233 +0000 UTC m=+3.102699823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret") pod "global-pull-secret-syncer-pzm5l" (UID: "a6273a00-72c1-456e-b47d-9a24d012f8c0") : object "kube-system"/"original-pull-secret" not registered Jul 03 10:05:31.565952 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564592 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bf44c15c-90ee-4d0e-afab-882c6620b5c5-device-dir\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.565952 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564723 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-kubelet\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.565952 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564746 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a8b64aea-2cb1-4919-ac4f-e002681cf606-env-overrides\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.565952 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564784 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-multus-conf-dir\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.566668 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564830 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bf44c15c-90ee-4d0e-afab-882c6620b5c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.566668 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564830 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a8b64aea-2cb1-4919-ac4f-e002681cf606-ovnkube-config\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.566668 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564843 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-host-var-lib-cni-bin\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.566668 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564855 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bf44c15c-90ee-4d0e-afab-882c6620b5c5-socket-dir\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.566668 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564874 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bf44c15c-90ee-4d0e-afab-882c6620b5c5-registration-dir\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.566668 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-host-run-multus-certs\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.566668 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564895 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-system-cni-dir\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.566668 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-cni-netd\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.566668 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564942 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-host-run-multus-certs\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.566668 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564951 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a8b64aea-2cb1-4919-ac4f-e002681cf606-ovnkube-script-lib\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.566668 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564985 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v84gz\" (UniqueName: \"kubernetes.io/projected/a8b64aea-2cb1-4919-ac4f-e002681cf606-kube-api-access-v84gz\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.566668 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.564994 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-cni-netd\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.566668 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.565010 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-os-release\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.566668 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.565038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-run-ovn-kubernetes\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.566668 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.565056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6273a00-72c1-456e-b47d-9a24d012f8c0-dbus\") pod \"global-pull-secret-syncer-pzm5l\" (UID: \"a6273a00-72c1-456e-b47d-9a24d012f8c0\") " pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:31.567425 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.565073 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-multus-socket-dir-parent\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.567425 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.565114 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8b64aea-2cb1-4919-ac4f-e002681cf606-host-run-ovn-kubernetes\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.567425 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.565135 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a6273a00-72c1-456e-b47d-9a24d012f8c0-dbus\") pod \"global-pull-secret-syncer-pzm5l\" (UID: \"a6273a00-72c1-456e-b47d-9a24d012f8c0\") " pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:31.567425 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.565149 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-host-var-lib-cni-multus\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.567425 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.565176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-host-var-lib-kubelet\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.567425 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.565197 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-os-release\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.567425 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.565259 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-multus-socket-dir-parent\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.567425 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.565263 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-host-var-lib-kubelet\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.567425 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.565294 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94799f7a-6383-48ca-8ba2-6d5d0eee3148-host-var-lib-cni-multus\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.567425 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.565331 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94799f7a-6383-48ca-8ba2-6d5d0eee3148-cni-binary-copy\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.567425 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.565477 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a8b64aea-2cb1-4919-ac4f-e002681cf606-ovnkube-script-lib\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.568354 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.568333 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a8b64aea-2cb1-4919-ac4f-e002681cf606-ovn-node-metrics-cert\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.573606 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:31.573580 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 10:05:31.573778 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:31.573610 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 10:05:31.573778 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:31.573625 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vbd6f for pod openshift-network-diagnostics/network-check-target-pzhjd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 10:05:31.573778 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:31.573754 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f podName:abbde693-f1f4-469f-bc67-c793cb2007de nodeName:}" failed. No retries permitted until 2026-07-03 10:05:32.073737214 +0000 UTC m=+3.111883803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vbd6f" (UniqueName: "kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f") pod "network-check-target-pzhjd" (UID: "abbde693-f1f4-469f-bc67-c793cb2007de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 10:05:31.579389 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.577255 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4jdc\" (UniqueName: \"kubernetes.io/projected/94799f7a-6383-48ca-8ba2-6d5d0eee3148-kube-api-access-m4jdc\") pod \"multus-fbl2t\" (UID: \"94799f7a-6383-48ca-8ba2-6d5d0eee3148\") " pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.579389 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.579316 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxh6v\" (UniqueName: \"kubernetes.io/projected/bf44c15c-90ee-4d0e-afab-882c6620b5c5-kube-api-access-bxh6v\") pod \"aws-ebs-csi-driver-node-wzpjg\" (UID: \"bf44c15c-90ee-4d0e-afab-882c6620b5c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.580760 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.580018 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v84gz\" (UniqueName: \"kubernetes.io/projected/a8b64aea-2cb1-4919-ac4f-e002681cf606-kube-api-access-v84gz\") pod \"ovnkube-node-knz5k\" (UID: \"a8b64aea-2cb1-4919-ac4f-e002681cf606\") " pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.648331 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.648292 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vbnml" Jul 03 10:05:31.660035 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:31.660004 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85cda35a_33a5_4f89_96f4_a2b3f8b585f7.slice/crio-7dd1322c0c9c8fb206e5b2d81aa1db8c1e3cd9d585b02883ec08d042e3a8a40b WatchSource:0}: Error finding container 7dd1322c0c9c8fb206e5b2d81aa1db8c1e3cd9d585b02883ec08d042e3a8a40b: Status 404 returned error can't find the container with id 7dd1322c0c9c8fb206e5b2d81aa1db8c1e3cd9d585b02883ec08d042e3a8a40b Jul 03 10:05:31.701459 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.701426 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8rwtr" Jul 03 10:05:31.714274 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.714021 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ktznn" Jul 03 10:05:31.721463 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.721216 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4hpsh" Jul 03 10:05:31.725965 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:31.725935 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4d4fbb_63d8_4ca2_aa9b_78b1c4076aaf.slice/crio-7c304e81d9362c1de04f0f57fbf5ecffc9ae0159a68ade62d9bb5ced6a00e60e WatchSource:0}: Error finding container 7c304e81d9362c1de04f0f57fbf5ecffc9ae0159a68ade62d9bb5ced6a00e60e: Status 404 returned error can't find the container with id 7c304e81d9362c1de04f0f57fbf5ecffc9ae0159a68ade62d9bb5ced6a00e60e Jul 03 10:05:31.730110 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.729973 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dbhqx" Jul 03 10:05:31.733582 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:31.733557 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8254340_be1e_449d_8022_1d4da25997ff.slice/crio-93af47d3c2529aba7f0a5efbb4dd43d9b8ade1f422516cd549e11f3ca1bdce64 WatchSource:0}: Error finding container 93af47d3c2529aba7f0a5efbb4dd43d9b8ade1f422516cd549e11f3ca1bdce64: Status 404 returned error can't find the container with id 93af47d3c2529aba7f0a5efbb4dd43d9b8ade1f422516cd549e11f3ca1bdce64 Jul 03 10:05:31.740457 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.740307 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qskxt" Jul 03 10:05:31.740619 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:31.740586 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ff74d6_895a_4a92_9da8_b13c8db95a92.slice/crio-04df9e699908ba3d2e1cce414cf4f0a4fc84bd9df7a7e39bc07641518d80f78b WatchSource:0}: Error finding container 04df9e699908ba3d2e1cce414cf4f0a4fc84bd9df7a7e39bc07641518d80f78b: Status 404 returned error can't find the container with id 04df9e699908ba3d2e1cce414cf4f0a4fc84bd9df7a7e39bc07641518d80f78b Jul 03 10:05:31.748092 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.748071 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:31.748521 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:31.748491 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e7cb03_364e_4480_84c8_e163592cfa64.slice/crio-af85b35238e37a21c880ed74b7cc74d35b5d6370f1585ba4235a538c7035be41 WatchSource:0}: Error finding container af85b35238e37a21c880ed74b7cc74d35b5d6370f1585ba4235a538c7035be41: Status 404 returned error can't find the container with id af85b35238e37a21c880ed74b7cc74d35b5d6370f1585ba4235a538c7035be41 Jul 03 10:05:31.751294 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.751275 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 10:05:31.756329 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.756308 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" Jul 03 10:05:31.758508 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:05:31.758478 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b64aea_2cb1_4919_ac4f_e002681cf606.slice/crio-3add120a12d3c60b0c36b70a3b9150a022cee3f6fddffe582a33d4eb11c879b2 WatchSource:0}: Error finding container 3add120a12d3c60b0c36b70a3b9150a022cee3f6fddffe582a33d4eb11c879b2: Status 404 returned error can't find the container with id 3add120a12d3c60b0c36b70a3b9150a022cee3f6fddffe582a33d4eb11c879b2 Jul 03 10:05:31.765555 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.765387 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fbl2t" Jul 03 10:05:31.968885 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:31.968845 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs\") pod \"network-metrics-daemon-lwkn9\" (UID: \"de5822f2-b52a-4283-938f-0395efeae942\") " pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:31.969059 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:31.968991 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 10:05:31.969059 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:31.969051 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs podName:de5822f2-b52a-4283-938f-0395efeae942 nodeName:}" failed. No retries permitted until 2026-07-03 10:05:32.969035365 +0000 UTC m=+4.007181952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs") pod "network-metrics-daemon-lwkn9" (UID: "de5822f2-b52a-4283-938f-0395efeae942") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 10:05:32.069692 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:32.069595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret\") pod \"global-pull-secret-syncer-pzm5l\" (UID: \"a6273a00-72c1-456e-b47d-9a24d012f8c0\") " pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:32.069855 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:32.069740 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 10:05:32.069855 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:32.069808 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret podName:a6273a00-72c1-456e-b47d-9a24d012f8c0 nodeName:}" failed. No retries permitted until 2026-07-03 10:05:33.069790883 +0000 UTC m=+4.107937488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret") pod "global-pull-secret-syncer-pzm5l" (UID: "a6273a00-72c1-456e-b47d-9a24d012f8c0") : object "kube-system"/"original-pull-secret" not registered Jul 03 10:05:32.170287 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:32.170243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbd6f\" (UniqueName: \"kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f\") pod \"network-check-target-pzhjd\" (UID: \"abbde693-f1f4-469f-bc67-c793cb2007de\") " pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:32.170468 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:32.170446 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 10:05:32.170565 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:32.170473 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 10:05:32.170565 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:32.170484 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vbd6f for pod openshift-network-diagnostics/network-check-target-pzhjd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 10:05:32.170565 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:32.170563 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f podName:abbde693-f1f4-469f-bc67-c793cb2007de nodeName:}" failed. No retries permitted until 2026-07-03 10:05:33.170533838 +0000 UTC m=+4.208680424 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbd6f" (UniqueName: "kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f") pod "network-check-target-pzhjd" (UID: "abbde693-f1f4-469f-bc67-c793cb2007de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 10:05:32.381637 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:32.381539 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-02 10:00:31 +0000 UTC" deadline="2028-04-05 18:15:58.278261954 +0000 UTC" Jul 03 10:05:32.381637 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:32.381601 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15416h10m25.896666765s" Jul 03 10:05:32.525647 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:32.525619 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:32.525816 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:32.525753 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzhjd" podUID="abbde693-f1f4-469f-bc67-c793cb2007de" Jul 03 10:05:32.525901 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:32.525809 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:32.526001 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:32.525960 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwkn9" podUID="de5822f2-b52a-4283-938f-0395efeae942" Jul 03 10:05:32.535351 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:32.535306 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" event={"ID":"a8b64aea-2cb1-4919-ac4f-e002681cf606","Type":"ContainerStarted","Data":"3add120a12d3c60b0c36b70a3b9150a022cee3f6fddffe582a33d4eb11c879b2"} Jul 03 10:05:32.536737 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:32.536708 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qskxt" event={"ID":"b6e7cb03-364e-4480-84c8-e163592cfa64","Type":"ContainerStarted","Data":"af85b35238e37a21c880ed74b7cc74d35b5d6370f1585ba4235a538c7035be41"} Jul 03 10:05:32.538136 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:32.538086 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ktznn" event={"ID":"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf","Type":"ContainerStarted","Data":"7c304e81d9362c1de04f0f57fbf5ecffc9ae0159a68ade62d9bb5ced6a00e60e"} Jul 03 10:05:32.539385 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:32.539347 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dbhqx" event={"ID":"21ff74d6-895a-4a92-9da8-b13c8db95a92","Type":"ContainerStarted","Data":"04df9e699908ba3d2e1cce414cf4f0a4fc84bd9df7a7e39bc07641518d80f78b"} Jul 03 10:05:32.540688 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:32.540654 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4hpsh" event={"ID":"f8254340-be1e-449d-8022-1d4da25997ff","Type":"ContainerStarted","Data":"93af47d3c2529aba7f0a5efbb4dd43d9b8ade1f422516cd549e11f3ca1bdce64"} Jul 03 10:05:32.542179 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:32.542154 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8rwtr" event={"ID":"ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d","Type":"ContainerStarted","Data":"75e742535c26d720d4c77d39b9507f5bac110525a6da07e6f36b4b29280e7838"} Jul 03 10:05:32.543563 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:32.543543 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vbnml" event={"ID":"85cda35a-33a5-4f89-96f4-a2b3f8b585f7","Type":"ContainerStarted","Data":"7dd1322c0c9c8fb206e5b2d81aa1db8c1e3cd9d585b02883ec08d042e3a8a40b"} Jul 03 10:05:32.977933 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:32.977900 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs\") pod \"network-metrics-daemon-lwkn9\" (UID: \"de5822f2-b52a-4283-938f-0395efeae942\") " pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:32.978124 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:32.978104 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 10:05:32.978213 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:32.978179 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs podName:de5822f2-b52a-4283-938f-0395efeae942 nodeName:}" failed. No retries permitted until 2026-07-03 10:05:34.978157206 +0000 UTC m=+6.016303806 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs") pod "network-metrics-daemon-lwkn9" (UID: "de5822f2-b52a-4283-938f-0395efeae942") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 10:05:33.078895 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:33.078630 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret\") pod \"global-pull-secret-syncer-pzm5l\" (UID: \"a6273a00-72c1-456e-b47d-9a24d012f8c0\") " pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:33.079044 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:33.078993 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 10:05:33.079106 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:33.079051 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret podName:a6273a00-72c1-456e-b47d-9a24d012f8c0 nodeName:}" failed. No retries permitted until 2026-07-03 10:05:35.079031544 +0000 UTC m=+6.117178136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret") pod "global-pull-secret-syncer-pzm5l" (UID: "a6273a00-72c1-456e-b47d-9a24d012f8c0") : object "kube-system"/"original-pull-secret" not registered Jul 03 10:05:33.180021 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:33.179898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbd6f\" (UniqueName: \"kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f\") pod \"network-check-target-pzhjd\" (UID: \"abbde693-f1f4-469f-bc67-c793cb2007de\") " pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:33.180190 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:33.180126 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 10:05:33.180190 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:33.180150 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 10:05:33.180190 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:33.180164 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vbd6f for pod openshift-network-diagnostics/network-check-target-pzhjd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 10:05:33.180368 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:33.180223 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f podName:abbde693-f1f4-469f-bc67-c793cb2007de nodeName:}" failed. No retries permitted until 2026-07-03 10:05:35.180205101 +0000 UTC m=+6.218351691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbd6f" (UniqueName: "kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f") pod "network-check-target-pzhjd" (UID: "abbde693-f1f4-469f-bc67-c793cb2007de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 10:05:33.382399 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:33.382287 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-02 10:00:31 +0000 UTC" deadline="2027-12-18 01:56:11.238322985 +0000 UTC" Jul 03 10:05:33.382399 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:33.382364 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12783h50m37.855963196s" Jul 03 10:05:33.527829 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:33.527211 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:33.527829 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:33.527421 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzm5l" podUID="a6273a00-72c1-456e-b47d-9a24d012f8c0" Jul 03 10:05:33.555305 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:33.555248 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" event={"ID":"bf44c15c-90ee-4d0e-afab-882c6620b5c5","Type":"ContainerStarted","Data":"49658237fc4264af0e3e955ffe9f95ecc21bef25284a08b081b5759783336b0d"} Jul 03 10:05:33.559431 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:33.559397 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.ec2.internal" event={"ID":"6b144ac3fd0711db4037382f38ebdd87","Type":"ContainerStarted","Data":"3950a9e8ca152620bff73991bd16e8ecc11f9a42839904669da08df1875102a6"} Jul 03 10:05:33.567138 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:33.567060 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fbl2t" event={"ID":"94799f7a-6383-48ca-8ba2-6d5d0eee3148","Type":"ContainerStarted","Data":"92c455f8fc9bf133998cd113e13f03fc626be626a220c75a0fb3f003414b8ddc"} Jul 03 10:05:34.524661 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:34.524629 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:34.525103 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:34.524786 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwkn9" podUID="de5822f2-b52a-4283-938f-0395efeae942" Jul 03 10:05:34.525280 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:34.525183 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:34.525390 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:34.525302 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzhjd" podUID="abbde693-f1f4-469f-bc67-c793cb2007de" Jul 03 10:05:34.581110 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:34.581072 2579 generic.go:356] "Generic (PLEG): container finished" podID="51aae6dcfb54935c8a4ffb1745f8057f" containerID="b1090656aa70307aa0ce915cbfee231f21cdbdd172492814cb9c1dced9dbf06c" exitCode=0 Jul 03 10:05:34.582014 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:34.581987 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal" event={"ID":"51aae6dcfb54935c8a4ffb1745f8057f","Type":"ContainerDied","Data":"b1090656aa70307aa0ce915cbfee231f21cdbdd172492814cb9c1dced9dbf06c"} Jul 03 10:05:34.597300 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:34.597242 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-105.ec2.internal" podStartSLOduration=4.597208532 podStartE2EDuration="4.597208532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 10:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:05:33.578077115 +0000 UTC m=+4.616223726" watchObservedRunningTime="2026-07-03 10:05:34.597208532 +0000 UTC m=+5.635355139" Jul 03 10:05:34.998338 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:34.998258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs\") pod \"network-metrics-daemon-lwkn9\" (UID: \"de5822f2-b52a-4283-938f-0395efeae942\") " pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:34.998500 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:34.998426 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 10:05:34.998500 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:34.998483 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs podName:de5822f2-b52a-4283-938f-0395efeae942 nodeName:}" failed. No retries permitted until 2026-07-03 10:05:38.998465912 +0000 UTC m=+10.036612500 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs") pod "network-metrics-daemon-lwkn9" (UID: "de5822f2-b52a-4283-938f-0395efeae942") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 10:05:35.099031 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:35.098995 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret\") pod \"global-pull-secret-syncer-pzm5l\" (UID: \"a6273a00-72c1-456e-b47d-9a24d012f8c0\") " pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:35.099255 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:35.099112 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 10:05:35.099255 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:35.099158 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret podName:a6273a00-72c1-456e-b47d-9a24d012f8c0 nodeName:}" failed. No retries permitted until 2026-07-03 10:05:39.099143519 +0000 UTC m=+10.137290105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret") pod "global-pull-secret-syncer-pzm5l" (UID: "a6273a00-72c1-456e-b47d-9a24d012f8c0") : object "kube-system"/"original-pull-secret" not registered Jul 03 10:05:35.200185 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:35.200142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbd6f\" (UniqueName: \"kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f\") pod \"network-check-target-pzhjd\" (UID: \"abbde693-f1f4-469f-bc67-c793cb2007de\") " pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:35.200376 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:35.200309 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 10:05:35.200376 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:35.200327 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 10:05:35.200376 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:35.200339 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vbd6f for pod openshift-network-diagnostics/network-check-target-pzhjd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 10:05:35.200539 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:35.200392 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f podName:abbde693-f1f4-469f-bc67-c793cb2007de nodeName:}" failed. No retries permitted until 2026-07-03 10:05:39.200375382 +0000 UTC m=+10.238521976 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbd6f" (UniqueName: "kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f") pod "network-check-target-pzhjd" (UID: "abbde693-f1f4-469f-bc67-c793cb2007de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 10:05:35.525618 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:35.525585 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:35.526081 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:35.525723 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzm5l" podUID="a6273a00-72c1-456e-b47d-9a24d012f8c0" Jul 03 10:05:36.525573 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:36.525532 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:36.525746 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:36.525663 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzhjd" podUID="abbde693-f1f4-469f-bc67-c793cb2007de" Jul 03 10:05:36.526109 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:36.525883 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:36.526109 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:36.525988 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwkn9" podUID="de5822f2-b52a-4283-938f-0395efeae942" Jul 03 10:05:37.525344 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:37.525263 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:37.525510 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:37.525397 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzm5l" podUID="a6273a00-72c1-456e-b47d-9a24d012f8c0" Jul 03 10:05:38.525676 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:38.525642 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:38.526111 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:38.525697 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:38.526111 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:38.525779 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzhjd" podUID="abbde693-f1f4-469f-bc67-c793cb2007de" Jul 03 10:05:38.526111 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:38.525952 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwkn9" podUID="de5822f2-b52a-4283-938f-0395efeae942" Jul 03 10:05:39.032208 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:39.032171 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs\") pod \"network-metrics-daemon-lwkn9\" (UID: \"de5822f2-b52a-4283-938f-0395efeae942\") " pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:39.032383 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:39.032324 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 10:05:39.032452 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:39.032406 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs podName:de5822f2-b52a-4283-938f-0395efeae942 nodeName:}" failed. No retries permitted until 2026-07-03 10:05:47.032384173 +0000 UTC m=+18.070530760 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs") pod "network-metrics-daemon-lwkn9" (UID: "de5822f2-b52a-4283-938f-0395efeae942") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 10:05:39.133634 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:39.133584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret\") pod \"global-pull-secret-syncer-pzm5l\" (UID: \"a6273a00-72c1-456e-b47d-9a24d012f8c0\") " pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:39.133820 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:39.133714 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 10:05:39.133820 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:39.133791 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret podName:a6273a00-72c1-456e-b47d-9a24d012f8c0 nodeName:}" failed. No retries permitted until 2026-07-03 10:05:47.133771418 +0000 UTC m=+18.171918007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret") pod "global-pull-secret-syncer-pzm5l" (UID: "a6273a00-72c1-456e-b47d-9a24d012f8c0") : object "kube-system"/"original-pull-secret" not registered Jul 03 10:05:39.234759 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:39.234715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbd6f\" (UniqueName: \"kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f\") pod \"network-check-target-pzhjd\" (UID: \"abbde693-f1f4-469f-bc67-c793cb2007de\") " pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:39.234951 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:39.234927 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 10:05:39.235034 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:39.234957 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 10:05:39.235034 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:39.234972 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vbd6f for pod openshift-network-diagnostics/network-check-target-pzhjd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 10:05:39.235152 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:39.235038 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f podName:abbde693-f1f4-469f-bc67-c793cb2007de nodeName:}" failed. No retries permitted until 2026-07-03 10:05:47.235017192 +0000 UTC m=+18.273163840 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbd6f" (UniqueName: "kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f") pod "network-check-target-pzhjd" (UID: "abbde693-f1f4-469f-bc67-c793cb2007de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 10:05:39.526239 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:39.526061 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:39.526239 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:39.526175 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzm5l" podUID="a6273a00-72c1-456e-b47d-9a24d012f8c0" Jul 03 10:05:40.525661 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:40.525627 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:40.525858 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:40.525762 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzhjd" podUID="abbde693-f1f4-469f-bc67-c793cb2007de" Jul 03 10:05:40.526316 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:40.526131 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:40.526316 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:40.526270 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwkn9" podUID="de5822f2-b52a-4283-938f-0395efeae942" Jul 03 10:05:41.525790 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:41.525585 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:41.525790 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:41.525716 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzm5l" podUID="a6273a00-72c1-456e-b47d-9a24d012f8c0" Jul 03 10:05:42.524649 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:42.524607 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:42.525096 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:42.524608 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:42.525096 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:42.524712 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzhjd" podUID="abbde693-f1f4-469f-bc67-c793cb2007de" Jul 03 10:05:42.525096 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:42.524796 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwkn9" podUID="de5822f2-b52a-4283-938f-0395efeae942" Jul 03 10:05:43.524912 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:43.524879 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:43.525385 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:43.525008 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzm5l" podUID="a6273a00-72c1-456e-b47d-9a24d012f8c0" Jul 03 10:05:44.525173 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:44.525142 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:44.525685 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:44.525148 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:44.525685 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:44.525261 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzhjd" podUID="abbde693-f1f4-469f-bc67-c793cb2007de" Jul 03 10:05:44.525685 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:44.525351 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwkn9" podUID="de5822f2-b52a-4283-938f-0395efeae942" Jul 03 10:05:45.524784 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:45.524753 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:45.524956 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:45.524876 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzm5l" podUID="a6273a00-72c1-456e-b47d-9a24d012f8c0" Jul 03 10:05:46.524801 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:46.524769 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:46.525277 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:46.524769 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:46.525277 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:46.524859 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzhjd" podUID="abbde693-f1f4-469f-bc67-c793cb2007de" Jul 03 10:05:46.525277 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:46.524939 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwkn9" podUID="de5822f2-b52a-4283-938f-0395efeae942" Jul 03 10:05:47.094397 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:47.094298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs\") pod \"network-metrics-daemon-lwkn9\" (UID: \"de5822f2-b52a-4283-938f-0395efeae942\") " pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:47.094568 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:47.094475 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 10:05:47.094568 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:47.094556 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs podName:de5822f2-b52a-4283-938f-0395efeae942 nodeName:}" failed. No retries permitted until 2026-07-03 10:06:03.094535614 +0000 UTC m=+34.132682212 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs") pod "network-metrics-daemon-lwkn9" (UID: "de5822f2-b52a-4283-938f-0395efeae942") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 10:05:47.195061 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:47.195022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret\") pod \"global-pull-secret-syncer-pzm5l\" (UID: \"a6273a00-72c1-456e-b47d-9a24d012f8c0\") " pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:47.195238 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:47.195185 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 10:05:47.195299 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:47.195267 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret podName:a6273a00-72c1-456e-b47d-9a24d012f8c0 nodeName:}" failed. No retries permitted until 2026-07-03 10:06:03.195253464 +0000 UTC m=+34.233400053 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret") pod "global-pull-secret-syncer-pzm5l" (UID: "a6273a00-72c1-456e-b47d-9a24d012f8c0") : object "kube-system"/"original-pull-secret" not registered Jul 03 10:05:47.296092 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:47.296059 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbd6f\" (UniqueName: \"kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f\") pod \"network-check-target-pzhjd\" (UID: \"abbde693-f1f4-469f-bc67-c793cb2007de\") " pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:47.296267 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:47.296248 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 10:05:47.296336 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:47.296272 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 10:05:47.296336 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:47.296287 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vbd6f for pod openshift-network-diagnostics/network-check-target-pzhjd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 10:05:47.296418 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:47.296339 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f podName:abbde693-f1f4-469f-bc67-c793cb2007de nodeName:}" failed. No retries permitted until 2026-07-03 10:06:03.296324925 +0000 UTC m=+34.334471510 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbd6f" (UniqueName: "kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f") pod "network-check-target-pzhjd" (UID: "abbde693-f1f4-469f-bc67-c793cb2007de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 10:05:47.525327 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:47.525294 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:47.525727 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:47.525435 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzm5l" podUID="a6273a00-72c1-456e-b47d-9a24d012f8c0" Jul 03 10:05:48.525225 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:48.525192 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:48.525395 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:48.525192 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:48.525395 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:48.525346 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwkn9" podUID="de5822f2-b52a-4283-938f-0395efeae942" Jul 03 10:05:48.525790 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:48.525437 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzhjd" podUID="abbde693-f1f4-469f-bc67-c793cb2007de" Jul 03 10:05:49.526261 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:49.526208 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:49.526720 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:49.526323 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzm5l" podUID="a6273a00-72c1-456e-b47d-9a24d012f8c0" Jul 03 10:05:50.525207 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:50.525170 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:50.525207 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:50.525193 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:50.525421 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:50.525311 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzhjd" podUID="abbde693-f1f4-469f-bc67-c793cb2007de" Jul 03 10:05:50.525480 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:50.525458 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwkn9" podUID="de5822f2-b52a-4283-938f-0395efeae942" Jul 03 10:05:51.525565 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:51.525241 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:51.526358 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:51.525681 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzm5l" podUID="a6273a00-72c1-456e-b47d-9a24d012f8c0" Jul 03 10:05:51.617701 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:51.617665 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4hpsh" event={"ID":"f8254340-be1e-449d-8022-1d4da25997ff","Type":"ContainerStarted","Data":"ada2a4a0626700fa545a4d5ff2547df0a79e34786d4f569559d77dc23579baa6"} Jul 03 10:05:51.619062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:51.619039 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vbnml" event={"ID":"85cda35a-33a5-4f89-96f4-a2b3f8b585f7","Type":"ContainerStarted","Data":"1d30975cdf1c06a16106e5ebe28e98c9fc38a47716ff3d6b455cb4b5811d17ab"} Jul 03 10:05:51.620445 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:51.620418 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" event={"ID":"bf44c15c-90ee-4d0e-afab-882c6620b5c5","Type":"ContainerStarted","Data":"efa61fdf4e3e8951851878e6c5e8249e0818cdfd8e4e0da4816679d8e7f762a1"} Jul 03 10:05:51.622361 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:51.622343 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal" event={"ID":"51aae6dcfb54935c8a4ffb1745f8057f","Type":"ContainerStarted","Data":"d50bd6e6500eb394734fef120ebd4263c07e8875c7d4c9a880ebbd0aa8335fd2"} Jul 03 10:05:51.624945 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:51.624925 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" event={"ID":"a8b64aea-2cb1-4919-ac4f-e002681cf606","Type":"ContainerStarted","Data":"8ebb53285785d98df3391894b7bfd754a768b05cb28a96efa34500a032d0c57b"} Jul 03 10:05:51.625041 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:51.625006 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" event={"ID":"a8b64aea-2cb1-4919-ac4f-e002681cf606","Type":"ContainerStarted","Data":"e40edb238fec3ec64556d959e73a93fd8941437e212580cecba48ddd80c659b1"} Jul 03 10:05:51.625106 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:51.625051 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" event={"ID":"a8b64aea-2cb1-4919-ac4f-e002681cf606","Type":"ContainerStarted","Data":"15a051780f833ab9d490c91379acf8f739cfa7b456829f8ca478a8b1690c415a"} Jul 03 10:05:51.625106 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:51.625093 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" event={"ID":"a8b64aea-2cb1-4919-ac4f-e002681cf606","Type":"ContainerStarted","Data":"a3624bc56e6a0fa29497750f4986a35d73ddcb1034bd070b3f5e29f31577161b"} Jul 03 10:05:51.626318 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:51.626292 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qskxt" event={"ID":"b6e7cb03-364e-4480-84c8-e163592cfa64","Type":"ContainerStarted","Data":"6909d94018c71eb0aa4ab9d3e0a4ea37737b73e85588f78e91bf59cda41856c8"} Jul 03 10:05:51.627829 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:51.627799 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ktznn" event={"ID":"3e4d4fbb-63d8-4ca2-aa9b-78b1c4076aaf","Type":"ContainerStarted","Data":"21611c11c57f2047fc92330f1254561e0089a41b18e853ef22550cc81cf5c118"} Jul 03 10:05:51.629469 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:51.629445 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fbl2t" event={"ID":"94799f7a-6383-48ca-8ba2-6d5d0eee3148","Type":"ContainerStarted","Data":"c9b40f57bfea33e7984b209236220691352901c01a9bfcc7a4778971a9bbb3fc"} Jul 03 10:05:51.631321 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:51.631248 2579 generic.go:356] "Generic (PLEG): container finished" podID="21ff74d6-895a-4a92-9da8-b13c8db95a92" containerID="ec7f271f319e76a6fec95d88e59163acc10cfb6289e9739d6c249565d9c48c76" exitCode=0 Jul 03 10:05:51.631321 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:51.631282 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dbhqx" event={"ID":"21ff74d6-895a-4a92-9da8-b13c8db95a92","Type":"ContainerDied","Data":"ec7f271f319e76a6fec95d88e59163acc10cfb6289e9739d6c249565d9c48c76"} Jul 03 10:05:51.632122 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:51.632007 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4hpsh" podStartSLOduration=3.583170675 podStartE2EDuration="22.631990944s" totalImagesPullingTime="19.048820269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 10:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:05:51.631615046 +0000 UTC m=+22.669761656" watchObservedRunningTime="2026-07-03 10:05:51.631990944 +0000 UTC m=+22.670137553" Jul 03 10:05:51.684653 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:51.684607 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qskxt" podStartSLOduration=3.6495021100000002 podStartE2EDuration="22.684593572s" totalImagesPullingTime="19.035091462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:05:51.668614376 +0000 UTC m=+22.706760982" watchObservedRunningTime="2026-07-03 10:05:51.684593572 +0000 UTC m=+22.722740179" Jul 03 10:05:51.684785 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:51.684755 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fbl2t" podStartSLOduration=4.505074319 podStartE2EDuration="22.684748337s" totalImagesPullingTime="18.179674018s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 10:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:05:51.684412877 +0000 UTC m=+22.722559503" watchObservedRunningTime="2026-07-03 10:05:51.684748337 +0000 UTC m=+22.722894944" Jul 03 10:05:51.702600 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:51.702556 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vbnml" podStartSLOduration=3.579397358 podStartE2EDuration="22.70254137s" totalImagesPullingTime="19.123144012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 10:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:05:51.701998266 +0000 UTC m=+22.740144872" watchObservedRunningTime="2026-07-03 10:05:51.70254137 +0000 UTC m=+22.740687977" Jul 03 10:05:51.722539 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:51.722491 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ktznn" podStartSLOduration=3.662800464 podStartE2EDuration="22.722472382s" totalImagesPullingTime="19.059671918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 10:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:05:51.722259898 +0000 UTC m=+22.760406506" watchObservedRunningTime="2026-07-03 10:05:51.722472382 +0000 UTC m=+22.760618991" Jul 03 10:05:52.371083 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:52.371058 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 03 10:05:52.421164 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:52.421062 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-03T10:05:52.371082218Z","UUID":"addd4daf-9c48-4cfa-a3ad-6bbd640d6797","Handler":null,"Name":"","Endpoint":""} Jul 03 10:05:52.422416 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:52.422399 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 03 10:05:52.422516 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:52.422440 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 03 10:05:52.524944 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:52.524863 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:52.525104 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:52.525008 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwkn9" podUID="de5822f2-b52a-4283-938f-0395efeae942" Jul 03 10:05:52.525104 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:52.525037 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:52.525195 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:52.525112 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzhjd" podUID="abbde693-f1f4-469f-bc67-c793cb2007de" Jul 03 10:05:52.634502 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:52.634468 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8rwtr" event={"ID":"ac9c2c8d-95e3-4f16-a4a4-fc2fa0fb359d","Type":"ContainerStarted","Data":"1ce645985dcf53112d97beade497cf520c7adf57498ce90e4821336a356333ff"} Jul 03 10:05:52.636365 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:52.636337 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" event={"ID":"bf44c15c-90ee-4d0e-afab-882c6620b5c5","Type":"ContainerStarted","Data":"f9e9323797a3130c88ec0a42e89a83e01048e0d4c19629aa1dd0e8091ef276c6"} Jul 03 10:05:52.639156 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:52.639123 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" event={"ID":"a8b64aea-2cb1-4919-ac4f-e002681cf606","Type":"ContainerStarted","Data":"bd39d61ecad65e588fb0c5473b2ed1888b228ef8f27117d7c095e395f90201d4"} Jul 03 10:05:52.639273 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:52.639200 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" event={"ID":"a8b64aea-2cb1-4919-ac4f-e002681cf606","Type":"ContainerStarted","Data":"a1798d640408670f3bf3413659f6a46ce1ba31af3e1e2a367868e1373d725f53"} Jul 03 10:05:52.648843 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:52.648806 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-105.ec2.internal" podStartSLOduration=22.648793379 podStartE2EDuration="22.648793379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 10:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:05:51.737796863 +0000 UTC m=+22.775943471" watchObservedRunningTime="2026-07-03 10:05:52.648793379 +0000 UTC m=+23.686939985" Jul 03 10:05:53.525033 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:53.525005 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:53.525192 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:53.525109 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzm5l" podUID="a6273a00-72c1-456e-b47d-9a24d012f8c0" Jul 03 10:05:53.609195 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:53.609163 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qskxt" Jul 03 10:05:53.609830 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:53.609814 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qskxt" Jul 03 10:05:53.624482 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:53.624428 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8rwtr" podStartSLOduration=5.554570995 podStartE2EDuration="24.624412969s" totalImagesPullingTime="19.069841974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 10:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:05:52.648348613 +0000 UTC m=+23.686495219" watchObservedRunningTime="2026-07-03 10:05:53.624412969 +0000 UTC m=+24.662559577" Jul 03 10:05:53.641462 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:53.641424 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qskxt" Jul 03 10:05:53.641876 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:53.641835 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qskxt" Jul 03 10:05:54.525663 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:54.525436 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:54.525813 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:54.525437 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:54.525813 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:54.525682 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzhjd" podUID="abbde693-f1f4-469f-bc67-c793cb2007de" Jul 03 10:05:54.525813 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:54.525761 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwkn9" podUID="de5822f2-b52a-4283-938f-0395efeae942" Jul 03 10:05:54.646827 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:54.646782 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" event={"ID":"a8b64aea-2cb1-4919-ac4f-e002681cf606","Type":"ContainerStarted","Data":"fb0227e18e24ef9a46cb97475b6747f539a6cbe98fe31ad9501d791de9943226"} Jul 03 10:05:54.648543 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:54.648521 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" event={"ID":"bf44c15c-90ee-4d0e-afab-882c6620b5c5","Type":"ContainerStarted","Data":"e62ae37cc4427d9501449763b3821cfc24d65bcbb647e6f2b0ae1e4a7f3536cf"} Jul 03 10:05:54.674924 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:54.674875 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wzpjg" podStartSLOduration=5.200335921 podStartE2EDuration="25.674858748s" totalImagesPullingTime="20.474522827s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 10:05:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:05:54.667714092 +0000 UTC m=+25.705860770" watchObservedRunningTime="2026-07-03 10:05:54.674858748 +0000 UTC m=+25.713005354" Jul 03 10:05:55.525437 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:55.525405 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:55.525625 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:55.525536 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzm5l" podUID="a6273a00-72c1-456e-b47d-9a24d012f8c0" Jul 03 10:05:56.525669 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:56.525591 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:56.525669 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:56.525645 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:56.526179 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:56.525739 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwkn9" podUID="de5822f2-b52a-4283-938f-0395efeae942" Jul 03 10:05:56.526179 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:56.525847 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzhjd" podUID="abbde693-f1f4-469f-bc67-c793cb2007de" Jul 03 10:05:57.525177 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:57.524926 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:57.525413 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:57.525236 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzm5l" podUID="a6273a00-72c1-456e-b47d-9a24d012f8c0" Jul 03 10:05:57.660986 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:57.660945 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" event={"ID":"a8b64aea-2cb1-4919-ac4f-e002681cf606","Type":"ContainerStarted","Data":"dbb63dd742c1956475e2c7e5c1a4d4e08d630d3d773f7b33deffaeffcb5f34d7"} Jul 03 10:05:57.661691 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:57.661241 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:57.662871 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:57.662838 2579 generic.go:356] "Generic (PLEG): container finished" podID="21ff74d6-895a-4a92-9da8-b13c8db95a92" containerID="87c5ab0c0602ce28d4283809a9f08ab38fa12edea56e49a6a200e0a3867a5080" exitCode=0 Jul 03 10:05:57.662871 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:57.662873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dbhqx" event={"ID":"21ff74d6-895a-4a92-9da8-b13c8db95a92","Type":"ContainerDied","Data":"87c5ab0c0602ce28d4283809a9f08ab38fa12edea56e49a6a200e0a3867a5080"} Jul 03 10:05:57.677704 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:57.677682 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:57.719439 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:57.719379 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" podStartSLOduration=9.379351436 podStartE2EDuration="28.719360645s" totalImagesPullingTime="19.340009209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 10:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:05:57.694835869 +0000 UTC m=+28.732982475" watchObservedRunningTime="2026-07-03 10:05:57.719360645 +0000 UTC m=+28.757507253" Jul 03 10:05:58.453783 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:58.453753 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:05:58.525403 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:58.525374 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:58.525579 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:58.525374 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:58.525579 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:58.525519 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwkn9" podUID="de5822f2-b52a-4283-938f-0395efeae942" Jul 03 10:05:58.525579 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:58.525564 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzhjd" podUID="abbde693-f1f4-469f-bc67-c793cb2007de" Jul 03 10:05:58.549212 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:58.549177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:05:58.666459 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:58.665822 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:58.666459 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:58.665909 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:58.684587 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:58.684562 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:05:59.214910 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:59.214668 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pzhjd"] Jul 03 10:05:59.215078 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:59.214994 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:05:59.215135 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:59.215102 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzhjd" podUID="abbde693-f1f4-469f-bc67-c793cb2007de" Jul 03 10:05:59.217746 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:59.217720 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lwkn9"] Jul 03 10:05:59.217863 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:59.217807 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:05:59.217903 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:59.217883 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwkn9" podUID="de5822f2-b52a-4283-938f-0395efeae942" Jul 03 10:05:59.227755 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:59.227725 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pzm5l"] Jul 03 10:05:59.227882 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:59.227830 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:05:59.227959 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:05:59.227938 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzm5l" podUID="a6273a00-72c1-456e-b47d-9a24d012f8c0" Jul 03 10:05:59.669104 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:59.669068 2579 generic.go:356] "Generic (PLEG): container finished" podID="21ff74d6-895a-4a92-9da8-b13c8db95a92" containerID="22956ab5ce298c943baec231b89722b4549d5c75226f7b292858314d7dd707db" exitCode=0 Jul 03 10:05:59.669486 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:05:59.669151 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dbhqx" event={"ID":"21ff74d6-895a-4a92-9da8-b13c8db95a92","Type":"ContainerDied","Data":"22956ab5ce298c943baec231b89722b4549d5c75226f7b292858314d7dd707db"} Jul 03 10:06:00.525137 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:00.525087 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:06:00.525337 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:00.525205 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzhjd" podUID="abbde693-f1f4-469f-bc67-c793cb2007de" Jul 03 10:06:00.525337 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:00.525251 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:06:00.525453 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:00.525388 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwkn9" podUID="de5822f2-b52a-4283-938f-0395efeae942" Jul 03 10:06:01.456274 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:01.456234 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:01.528213 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:01.528187 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:06:01.528390 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:01.528297 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzm5l" podUID="a6273a00-72c1-456e-b47d-9a24d012f8c0" Jul 03 10:06:01.557812 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:01.557783 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:01.675385 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:01.675355 2579 generic.go:356] "Generic (PLEG): container finished" podID="21ff74d6-895a-4a92-9da8-b13c8db95a92" containerID="5c522ebdde9dd47c3218731ba88be19dc7b183fea6999e994bd033ce5df677a4" exitCode=0 Jul 03 10:06:01.675385 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:01.675400 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dbhqx" event={"ID":"21ff74d6-895a-4a92-9da8-b13c8db95a92","Type":"ContainerDied","Data":"5c522ebdde9dd47c3218731ba88be19dc7b183fea6999e994bd033ce5df677a4"} Jul 03 10:06:02.525185 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:02.525152 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:06:02.525608 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:02.525281 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pzhjd" podUID="abbde693-f1f4-469f-bc67-c793cb2007de" Jul 03 10:06:02.525608 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:02.525340 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:06:02.525608 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:02.525463 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lwkn9" podUID="de5822f2-b52a-4283-938f-0395efeae942" Jul 03 10:06:03.117826 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.117790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs\") pod \"network-metrics-daemon-lwkn9\" (UID: \"de5822f2-b52a-4283-938f-0395efeae942\") " pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:06:03.117989 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:03.117909 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 10:06:03.117989 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:03.117968 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs podName:de5822f2-b52a-4283-938f-0395efeae942 nodeName:}" failed. No retries permitted until 2026-07-03 10:06:35.117951033 +0000 UTC m=+66.156097621 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs") pod "network-metrics-daemon-lwkn9" (UID: "de5822f2-b52a-4283-938f-0395efeae942") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 10:06:03.218738 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.218701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret\") pod \"global-pull-secret-syncer-pzm5l\" (UID: \"a6273a00-72c1-456e-b47d-9a24d012f8c0\") " pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:06:03.218916 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:03.218831 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 10:06:03.218916 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:03.218889 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret podName:a6273a00-72c1-456e-b47d-9a24d012f8c0 nodeName:}" failed. No retries permitted until 2026-07-03 10:06:35.218874666 +0000 UTC m=+66.257021251 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret") pod "global-pull-secret-syncer-pzm5l" (UID: "a6273a00-72c1-456e-b47d-9a24d012f8c0") : object "kube-system"/"original-pull-secret" not registered Jul 03 10:06:03.319611 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.319561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbd6f\" (UniqueName: \"kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f\") pod \"network-check-target-pzhjd\" (UID: \"abbde693-f1f4-469f-bc67-c793cb2007de\") " pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:06:03.319779 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:03.319729 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 10:06:03.319779 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:03.319756 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 10:06:03.319779 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:03.319766 2579 projected.go:196] Error preparing data for projected volume kube-api-access-vbd6f for pod openshift-network-diagnostics/network-check-target-pzhjd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 10:06:03.319908 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:03.319815 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f podName:abbde693-f1f4-469f-bc67-c793cb2007de nodeName:}" failed. No retries permitted until 2026-07-03 10:06:35.319801622 +0000 UTC m=+66.357948211 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbd6f" (UniqueName: "kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f") pod "network-check-target-pzhjd" (UID: "abbde693-f1f4-469f-bc67-c793cb2007de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 10:06:03.528327 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.528255 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:06:03.528713 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:03.528353 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pzm5l" podUID="a6273a00-72c1-456e-b47d-9a24d012f8c0" Jul 03 10:06:03.775329 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.775300 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-105.ec2.internal" event="NodeReady" Jul 03 10:06:03.775498 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.775443 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 03 10:06:03.833151 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.833055 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59748c45cf-vll4w"] Jul 03 10:06:03.835904 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.835880 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:03.838232 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.838190 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 03 10:06:03.838365 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.838194 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zmbwq\"" Jul 03 10:06:03.838365 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.838304 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 03 10:06:03.841464 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.841437 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 03 10:06:03.848455 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.848435 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 03 10:06:03.849150 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.849129 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qz8nn"] Jul 03 10:06:03.850785 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.850768 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qz8nn" Jul 03 10:06:03.861044 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.861024 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59748c45cf-vll4w"] Jul 03 10:06:03.861417 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.861401 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 03 10:06:03.861524 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.861507 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 03 10:06:03.861789 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.861772 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-sl8pb\"" Jul 03 10:06:03.861856 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.861804 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 03 10:06:03.861856 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.861806 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-78277"] Jul 03 10:06:03.863551 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.863535 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-78277" Jul 03 10:06:03.880492 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.880461 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 03 10:06:03.880934 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.880912 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sgbfl\"" Jul 03 10:06:03.881034 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.880975 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 03 10:06:03.885459 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.885436 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qz8nn"] Jul 03 10:06:03.892077 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.892055 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-78277"] Jul 03 10:06:03.924288 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.924262 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:03.924446 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.924292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0eac25e7-d923-45c0-98ca-69884664d7fc-installation-pull-secrets\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:03.924446 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.924321 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szpmp\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-kube-api-access-szpmp\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:03.924446 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.924345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0eac25e7-d923-45c0-98ca-69884664d7fc-image-registry-private-configuration\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:03.924446 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.924417 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-certificates\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:03.924625 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.924462 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0eac25e7-d923-45c0-98ca-69884664d7fc-ca-trust-extracted\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:03.924625 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.924494 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-bound-sa-token\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:03.924625 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:03.924564 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0eac25e7-d923-45c0-98ca-69884664d7fc-trusted-ca\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:04.025650 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.025613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0eac25e7-d923-45c0-98ca-69884664d7fc-ca-trust-extracted\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:04.025650 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.025655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-bound-sa-token\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:04.025897 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.025806 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0eac25e7-d923-45c0-98ca-69884664d7fc-trusted-ca\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:04.025897 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.025842 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xzpt\" (UniqueName: \"kubernetes.io/projected/30ca9240-21bd-4d28-85d7-5ac0d2f438bb-kube-api-access-9xzpt\") pod \"ingress-canary-qz8nn\" (UID: \"30ca9240-21bd-4d28-85d7-5ac0d2f438bb\") " pod="openshift-ingress-canary/ingress-canary-qz8nn" Jul 03 10:06:04.025897 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.025880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30ca9240-21bd-4d28-85d7-5ac0d2f438bb-cert\") pod \"ingress-canary-qz8nn\" (UID: \"30ca9240-21bd-4d28-85d7-5ac0d2f438bb\") " pod="openshift-ingress-canary/ingress-canary-qz8nn" Jul 03 10:06:04.026047 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.025907 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-config-volume\") pod \"dns-default-78277\" (UID: \"c6b0923e-598e-4d07-bbd0-cdf55b61a8af\") " pod="openshift-dns/dns-default-78277" Jul 03 10:06:04.026047 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.025966 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-tmp-dir\") pod \"dns-default-78277\" (UID: \"c6b0923e-598e-4d07-bbd0-cdf55b61a8af\") " pod="openshift-dns/dns-default-78277" Jul 03 10:06:04.026047 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.026004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dszdk\" (UniqueName: \"kubernetes.io/projected/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-kube-api-access-dszdk\") pod \"dns-default-78277\" (UID: \"c6b0923e-598e-4d07-bbd0-cdf55b61a8af\") " pod="openshift-dns/dns-default-78277" Jul 03 10:06:04.026361 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.026081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:04.026361 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.026121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0eac25e7-d923-45c0-98ca-69884664d7fc-installation-pull-secrets\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:04.026361 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.026125 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0eac25e7-d923-45c0-98ca-69884664d7fc-ca-trust-extracted\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:04.026361 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.026161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szpmp\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-kube-api-access-szpmp\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:04.026361 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:04.026262 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 03 10:06:04.026361 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:04.026285 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59748c45cf-vll4w: secret "image-registry-tls" not found Jul 03 10:06:04.026666 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:04.026373 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls podName:0eac25e7-d923-45c0-98ca-69884664d7fc nodeName:}" failed. No retries permitted until 2026-07-03 10:06:04.526342099 +0000 UTC m=+35.564488700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls") pod "image-registry-59748c45cf-vll4w" (UID: "0eac25e7-d923-45c0-98ca-69884664d7fc") : secret "image-registry-tls" not found Jul 03 10:06:04.026666 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.026463 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls\") pod \"dns-default-78277\" (UID: \"c6b0923e-598e-4d07-bbd0-cdf55b61a8af\") " pod="openshift-dns/dns-default-78277" Jul 03 10:06:04.026666 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.026501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0eac25e7-d923-45c0-98ca-69884664d7fc-image-registry-private-configuration\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:04.026666 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.026531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-certificates\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:04.027138 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.027109 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-certificates\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:04.027256 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.027228 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0eac25e7-d923-45c0-98ca-69884664d7fc-trusted-ca\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:04.030665 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.030645 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0eac25e7-d923-45c0-98ca-69884664d7fc-image-registry-private-configuration\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:04.030825 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.030806 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0eac25e7-d923-45c0-98ca-69884664d7fc-installation-pull-secrets\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:04.039414 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.039392 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-bound-sa-token\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:04.039877 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.039843 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szpmp\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-kube-api-access-szpmp\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:04.098822 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.098793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:04.127864 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.127830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xzpt\" (UniqueName: \"kubernetes.io/projected/30ca9240-21bd-4d28-85d7-5ac0d2f438bb-kube-api-access-9xzpt\") pod \"ingress-canary-qz8nn\" (UID: \"30ca9240-21bd-4d28-85d7-5ac0d2f438bb\") " pod="openshift-ingress-canary/ingress-canary-qz8nn" Jul 03 10:06:04.127991 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.127880 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30ca9240-21bd-4d28-85d7-5ac0d2f438bb-cert\") pod \"ingress-canary-qz8nn\" (UID: \"30ca9240-21bd-4d28-85d7-5ac0d2f438bb\") " pod="openshift-ingress-canary/ingress-canary-qz8nn" Jul 03 10:06:04.127991 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.127905 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-config-volume\") pod \"dns-default-78277\" (UID: \"c6b0923e-598e-4d07-bbd0-cdf55b61a8af\") " pod="openshift-dns/dns-default-78277" Jul 03 10:06:04.128116 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.128094 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-tmp-dir\") pod \"dns-default-78277\" (UID: \"c6b0923e-598e-4d07-bbd0-cdf55b61a8af\") " pod="openshift-dns/dns-default-78277" Jul 03 10:06:04.128173 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.128138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dszdk\" (UniqueName: \"kubernetes.io/projected/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-kube-api-access-dszdk\") pod \"dns-default-78277\" (UID: \"c6b0923e-598e-4d07-bbd0-cdf55b61a8af\") " pod="openshift-dns/dns-default-78277" Jul 03 10:06:04.128252 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.128213 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls\") pod \"dns-default-78277\" (UID: \"c6b0923e-598e-4d07-bbd0-cdf55b61a8af\") " pod="openshift-dns/dns-default-78277" Jul 03 10:06:04.128402 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:04.128380 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 10:06:04.128443 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.128422 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-tmp-dir\") pod \"dns-default-78277\" (UID: \"c6b0923e-598e-4d07-bbd0-cdf55b61a8af\") " pod="openshift-dns/dns-default-78277" Jul 03 10:06:04.128486 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:04.128476 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls podName:c6b0923e-598e-4d07-bbd0-cdf55b61a8af nodeName:}" failed. No retries permitted until 2026-07-03 10:06:04.628458127 +0000 UTC m=+35.666604730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls") pod "dns-default-78277" (UID: "c6b0923e-598e-4d07-bbd0-cdf55b61a8af") : secret "dns-default-metrics-tls" not found Jul 03 10:06:04.128531 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.128516 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-config-volume\") pod \"dns-default-78277\" (UID: \"c6b0923e-598e-4d07-bbd0-cdf55b61a8af\") " pod="openshift-dns/dns-default-78277" Jul 03 10:06:04.130513 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.130492 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30ca9240-21bd-4d28-85d7-5ac0d2f438bb-cert\") pod \"ingress-canary-qz8nn\" (UID: \"30ca9240-21bd-4d28-85d7-5ac0d2f438bb\") " pod="openshift-ingress-canary/ingress-canary-qz8nn" Jul 03 10:06:04.139007 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.138978 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xzpt\" (UniqueName: \"kubernetes.io/projected/30ca9240-21bd-4d28-85d7-5ac0d2f438bb-kube-api-access-9xzpt\") pod \"ingress-canary-qz8nn\" (UID: \"30ca9240-21bd-4d28-85d7-5ac0d2f438bb\") " pod="openshift-ingress-canary/ingress-canary-qz8nn" Jul 03 10:06:04.141148 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.141124 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dszdk\" (UniqueName: \"kubernetes.io/projected/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-kube-api-access-dszdk\") pod \"dns-default-78277\" (UID: \"c6b0923e-598e-4d07-bbd0-cdf55b61a8af\") " pod="openshift-dns/dns-default-78277" Jul 03 10:06:04.158364 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.158337 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qz8nn" Jul 03 10:06:04.200208 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.199951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:04.305761 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.305732 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qz8nn"] Jul 03 10:06:04.310170 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:06:04.310140 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ca9240_21bd_4d28_85d7_5ac0d2f438bb.slice/crio-604a68462a182d49f8267247b1650acbf5901c6dda1f4c928fbda088a2ebced2 WatchSource:0}: Error finding container 604a68462a182d49f8267247b1650acbf5901c6dda1f4c928fbda088a2ebced2: Status 404 returned error can't find the container with id 604a68462a182d49f8267247b1650acbf5901c6dda1f4c928fbda088a2ebced2 Jul 03 10:06:04.524883 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.524778 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:06:04.524883 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.524778 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:06:04.527605 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.527383 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 03 10:06:04.527605 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.527426 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 03 10:06:04.527605 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.527404 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 03 10:06:04.527605 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.527493 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xh5d7\"" Jul 03 10:06:04.527605 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.527404 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-njhrs\"" Jul 03 10:06:04.532178 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.532157 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:04.532655 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:04.532323 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 03 10:06:04.532655 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:04.532341 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59748c45cf-vll4w: secret "image-registry-tls" not found Jul 03 10:06:04.532655 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:04.532407 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls podName:0eac25e7-d923-45c0-98ca-69884664d7fc nodeName:}" failed. No retries permitted until 2026-07-03 10:06:05.53238553 +0000 UTC m=+36.570532118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls") pod "image-registry-59748c45cf-vll4w" (UID: "0eac25e7-d923-45c0-98ca-69884664d7fc") : secret "image-registry-tls" not found Jul 03 10:06:04.632676 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.632613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls\") pod \"dns-default-78277\" (UID: \"c6b0923e-598e-4d07-bbd0-cdf55b61a8af\") " pod="openshift-dns/dns-default-78277" Jul 03 10:06:04.632865 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:04.632748 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 10:06:04.632865 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:04.632826 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls podName:c6b0923e-598e-4d07-bbd0-cdf55b61a8af nodeName:}" failed. No retries permitted until 2026-07-03 10:06:05.632810214 +0000 UTC m=+36.670956799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls") pod "dns-default-78277" (UID: "c6b0923e-598e-4d07-bbd0-cdf55b61a8af") : secret "dns-default-metrics-tls" not found Jul 03 10:06:04.683567 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:04.683528 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qz8nn" event={"ID":"30ca9240-21bd-4d28-85d7-5ac0d2f438bb","Type":"ContainerStarted","Data":"604a68462a182d49f8267247b1650acbf5901c6dda1f4c928fbda088a2ebced2"} Jul 03 10:06:05.525069 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:05.525035 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:06:05.527617 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:05.527594 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 03 10:06:05.540044 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:05.540015 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:05.540415 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:05.540166 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 03 10:06:05.540415 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:05.540183 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59748c45cf-vll4w: secret "image-registry-tls" not found Jul 03 10:06:05.540415 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:05.540276 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls podName:0eac25e7-d923-45c0-98ca-69884664d7fc nodeName:}" failed. No retries permitted until 2026-07-03 10:06:07.540254424 +0000 UTC m=+38.578401009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls") pod "image-registry-59748c45cf-vll4w" (UID: "0eac25e7-d923-45c0-98ca-69884664d7fc") : secret "image-registry-tls" not found Jul 03 10:06:05.641395 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:05.641357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls\") pod \"dns-default-78277\" (UID: \"c6b0923e-598e-4d07-bbd0-cdf55b61a8af\") " pod="openshift-dns/dns-default-78277" Jul 03 10:06:05.641587 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:05.641508 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 10:06:05.641587 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:05.641576 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls podName:c6b0923e-598e-4d07-bbd0-cdf55b61a8af nodeName:}" failed. No retries permitted until 2026-07-03 10:06:07.641556262 +0000 UTC m=+38.679702853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls") pod "dns-default-78277" (UID: "c6b0923e-598e-4d07-bbd0-cdf55b61a8af") : secret "dns-default-metrics-tls" not found Jul 03 10:06:07.558465 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:07.558441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:07.558762 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:07.558586 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 03 10:06:07.558762 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:07.558602 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59748c45cf-vll4w: secret "image-registry-tls" not found Jul 03 10:06:07.558762 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:07.558652 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls podName:0eac25e7-d923-45c0-98ca-69884664d7fc nodeName:}" failed. No retries permitted until 2026-07-03 10:06:11.558638475 +0000 UTC m=+42.596785059 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls") pod "image-registry-59748c45cf-vll4w" (UID: "0eac25e7-d923-45c0-98ca-69884664d7fc") : secret "image-registry-tls" not found Jul 03 10:06:07.659848 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:07.659611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls\") pod \"dns-default-78277\" (UID: \"c6b0923e-598e-4d07-bbd0-cdf55b61a8af\") " pod="openshift-dns/dns-default-78277" Jul 03 10:06:07.659984 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:07.659762 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 10:06:07.660043 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:07.659988 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls podName:c6b0923e-598e-4d07-bbd0-cdf55b61a8af nodeName:}" failed. No retries permitted until 2026-07-03 10:06:11.659966657 +0000 UTC m=+42.698113245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls") pod "dns-default-78277" (UID: "c6b0923e-598e-4d07-bbd0-cdf55b61a8af") : secret "dns-default-metrics-tls" not found Jul 03 10:06:07.694006 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:07.693977 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dbhqx" event={"ID":"21ff74d6-895a-4a92-9da8-b13c8db95a92","Type":"ContainerStarted","Data":"9ceddf65e2d4b30858c8a80d4ec40b38b49635e774fc0534e56341de8e2dcf6e"} Jul 03 10:06:07.695460 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:07.695434 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qz8nn" event={"ID":"30ca9240-21bd-4d28-85d7-5ac0d2f438bb","Type":"ContainerStarted","Data":"b45eec72c89e0fd8b28888b4495e45c9259420d58e8829a56e8a66b9c31c6f5c"} Jul 03 10:06:07.737542 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:07.737415 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qz8nn" podStartSLOduration=1.581448897 podStartE2EDuration="4.73739393s" totalImagesPullingTime="3.155945033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:06:07.736705972 +0000 UTC m=+38.774852582" watchObservedRunningTime="2026-07-03 10:06:07.73739393 +0000 UTC m=+38.775540539" Jul 03 10:06:08.331160 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:08.331124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:08.434122 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:08.434071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:08.699799 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:08.699768 2579 generic.go:356] "Generic (PLEG): container finished" podID="21ff74d6-895a-4a92-9da8-b13c8db95a92" containerID="9ceddf65e2d4b30858c8a80d4ec40b38b49635e774fc0534e56341de8e2dcf6e" exitCode=0 Jul 03 10:06:08.700138 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:08.699866 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dbhqx" event={"ID":"21ff74d6-895a-4a92-9da8-b13c8db95a92","Type":"ContainerDied","Data":"9ceddf65e2d4b30858c8a80d4ec40b38b49635e774fc0534e56341de8e2dcf6e"} Jul 03 10:06:08.973874 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:08.973826 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:09.076260 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:09.076203 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:09.189277 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:09.189186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:09.300757 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:09.300728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:09.704768 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:09.704737 2579 generic.go:356] "Generic (PLEG): container finished" podID="21ff74d6-895a-4a92-9da8-b13c8db95a92" containerID="c73095b8af2179c3b87e47129f5eef009a332ea79d0a3b363840aed6537db40f" exitCode=0 Jul 03 10:06:09.705310 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:09.704786 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dbhqx" event={"ID":"21ff74d6-895a-4a92-9da8-b13c8db95a92","Type":"ContainerDied","Data":"c73095b8af2179c3b87e47129f5eef009a332ea79d0a3b363840aed6537db40f"} Jul 03 10:06:09.976252 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:09.976162 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6cgf4"] Jul 03 10:06:10.003176 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.003144 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6cgf4"] Jul 03 10:06:10.003329 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.003292 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:10.006265 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.006237 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 03 10:06:10.006387 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.006248 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 03 10:06:10.006435 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.006390 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f8d4v\"" Jul 03 10:06:10.006740 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.006721 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 03 10:06:10.006828 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.006724 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 03 10:06:10.026602 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.026559 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:10.079909 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.079868 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5570057e-5c51-449d-8390-7879cae4f8c8-data-volume\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:10.079909 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.079910 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5570057e-5c51-449d-8390-7879cae4f8c8-crio-socket\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:10.080097 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.079928 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5570057e-5c51-449d-8390-7879cae4f8c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:10.080097 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.080039 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4w6m\" (UniqueName: \"kubernetes.io/projected/5570057e-5c51-449d-8390-7879cae4f8c8-kube-api-access-b4w6m\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:10.080097 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.080083 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5570057e-5c51-449d-8390-7879cae4f8c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:10.124600 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.124568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:10.180961 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.180923 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5570057e-5c51-449d-8390-7879cae4f8c8-data-volume\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:10.180961 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.180965 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5570057e-5c51-449d-8390-7879cae4f8c8-crio-socket\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:10.181166 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.180981 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5570057e-5c51-449d-8390-7879cae4f8c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:10.181166 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.181044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4w6m\" (UniqueName: \"kubernetes.io/projected/5570057e-5c51-449d-8390-7879cae4f8c8-kube-api-access-b4w6m\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:10.181166 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.181083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5570057e-5c51-449d-8390-7879cae4f8c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:10.181166 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:10.181145 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 03 10:06:10.181378 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:10.181176 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 10:06:10.181378 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.181186 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5570057e-5c51-449d-8390-7879cae4f8c8-crio-socket\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:10.181378 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:10.181256 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5570057e-5c51-449d-8390-7879cae4f8c8-kube-rbac-proxy-cm podName:5570057e-5c51-449d-8390-7879cae4f8c8 nodeName:}" failed. No retries permitted until 2026-07-03 10:06:10.68121245 +0000 UTC m=+41.719359049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5570057e-5c51-449d-8390-7879cae4f8c8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6cgf4" (UID: "5570057e-5c51-449d-8390-7879cae4f8c8") : configmap "kube-rbac-proxy" not found Jul 03 10:06:10.181378 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:10.181292 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5570057e-5c51-449d-8390-7879cae4f8c8-insights-runtime-extractor-tls podName:5570057e-5c51-449d-8390-7879cae4f8c8 nodeName:}" failed. No retries permitted until 2026-07-03 10:06:10.681269782 +0000 UTC m=+41.719416384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5570057e-5c51-449d-8390-7879cae4f8c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6cgf4" (UID: "5570057e-5c51-449d-8390-7879cae4f8c8") : secret "insights-runtime-extractor-tls" not found Jul 03 10:06:10.181378 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.181324 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5570057e-5c51-449d-8390-7879cae4f8c8-data-volume\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:10.191657 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.191637 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4w6m\" (UniqueName: \"kubernetes.io/projected/5570057e-5c51-449d-8390-7879cae4f8c8-kube-api-access-b4w6m\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:10.230831 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.230760 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:10.340187 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.340148 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:10.445128 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.445099 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-h7ks5"] Jul 03 10:06:10.465525 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.465502 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-h7ks5" Jul 03 10:06:10.467196 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.467171 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:10.468634 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.468612 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 03 10:06:10.468747 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.468640 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-654lx\"" Jul 03 10:06:10.469048 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.469029 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 03 10:06:10.475741 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.475721 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-h7ks5"] Jul 03 10:06:10.482234 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.482162 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2w48\" (UniqueName: \"kubernetes.io/projected/f3f161fb-f33b-4aab-aa23-3a42315d24b0-kube-api-access-t2w48\") pod \"migrator-d57d58cc9-h7ks5\" (UID: \"f3f161fb-f33b-4aab-aa23-3a42315d24b0\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-h7ks5" Jul 03 10:06:10.568443 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.568412 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:10.582570 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.582547 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2w48\" (UniqueName: \"kubernetes.io/projected/f3f161fb-f33b-4aab-aa23-3a42315d24b0-kube-api-access-t2w48\") pod \"migrator-d57d58cc9-h7ks5\" (UID: \"f3f161fb-f33b-4aab-aa23-3a42315d24b0\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-h7ks5" Jul 03 10:06:10.590819 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.590791 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2w48\" (UniqueName: \"kubernetes.io/projected/f3f161fb-f33b-4aab-aa23-3a42315d24b0-kube-api-access-t2w48\") pod \"migrator-d57d58cc9-h7ks5\" (UID: \"f3f161fb-f33b-4aab-aa23-3a42315d24b0\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-h7ks5" Jul 03 10:06:10.670725 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.670693 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:10.682870 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.682847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5570057e-5c51-449d-8390-7879cae4f8c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:10.682927 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.682882 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5570057e-5c51-449d-8390-7879cae4f8c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:10.682978 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:10.682966 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 03 10:06:10.683009 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:10.682997 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 10:06:10.683038 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:10.683025 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5570057e-5c51-449d-8390-7879cae4f8c8-kube-rbac-proxy-cm podName:5570057e-5c51-449d-8390-7879cae4f8c8 nodeName:}" failed. No retries permitted until 2026-07-03 10:06:11.683010792 +0000 UTC m=+42.721157390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5570057e-5c51-449d-8390-7879cae4f8c8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6cgf4" (UID: "5570057e-5c51-449d-8390-7879cae4f8c8") : configmap "kube-rbac-proxy" not found Jul 03 10:06:10.683074 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:10.683045 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5570057e-5c51-449d-8390-7879cae4f8c8-insights-runtime-extractor-tls podName:5570057e-5c51-449d-8390-7879cae4f8c8 nodeName:}" failed. No retries permitted until 2026-07-03 10:06:11.683032772 +0000 UTC m=+42.721179357 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5570057e-5c51-449d-8390-7879cae4f8c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6cgf4" (UID: "5570057e-5c51-449d-8390-7879cae4f8c8") : secret "insights-runtime-extractor-tls" not found Jul 03 10:06:10.709867 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.709835 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dbhqx" event={"ID":"21ff74d6-895a-4a92-9da8-b13c8db95a92","Type":"ContainerStarted","Data":"efae9b9a106e389d8f5c944b1db481708a49d0120e447b7511575e860b2b7ea7"} Jul 03 10:06:10.734886 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.734798 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dbhqx" podStartSLOduration=9.149709102 podStartE2EDuration="41.734779776s" totalImagesPullingTime="32.585070674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 10:05:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:06:10.733617386 +0000 UTC m=+41.771763993" watchObservedRunningTime="2026-07-03 10:06:10.734779776 +0000 UTC m=+41.772926384" Jul 03 10:06:10.773725 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.773696 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-h7ks5" Jul 03 10:06:10.777576 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.777554 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:10.927629 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.927597 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-h7ks5"] Jul 03 10:06:10.930801 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:06:10.930770 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f161fb_f33b_4aab_aa23_3a42315d24b0.slice/crio-3706a7a7410a338d89d23316b48e2446bc7f77d20f3938f3a8f265007f05307e WatchSource:0}: Error finding container 3706a7a7410a338d89d23316b48e2446bc7f77d20f3938f3a8f265007f05307e: Status 404 returned error can't find the container with id 3706a7a7410a338d89d23316b48e2446bc7f77d20f3938f3a8f265007f05307e Jul 03 10:06:10.958787 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:10.958756 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:11.062057 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:11.061978 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:11.156983 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:11.156953 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:11.360434 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:11.360409 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4hpsh_f8254340-be1e-449d-8022-1d4da25997ff/dns-node-resolver/0.log" Jul 03 10:06:11.502498 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:11.502466 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:11.589482 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:11.589455 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:11.589613 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:11.589594 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 03 10:06:11.589663 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:11.589615 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59748c45cf-vll4w: secret "image-registry-tls" not found Jul 03 10:06:11.589701 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:11.589665 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls podName:0eac25e7-d923-45c0-98ca-69884664d7fc nodeName:}" failed. No retries permitted until 2026-07-03 10:06:19.589651481 +0000 UTC m=+50.627798066 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls") pod "image-registry-59748c45cf-vll4w" (UID: "0eac25e7-d923-45c0-98ca-69884664d7fc") : secret "image-registry-tls" not found Jul 03 10:06:11.603767 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:11.603738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:11.690230 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:11.690136 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls\") pod \"dns-default-78277\" (UID: \"c6b0923e-598e-4d07-bbd0-cdf55b61a8af\") " pod="openshift-dns/dns-default-78277" Jul 03 10:06:11.690230 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:11.690173 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5570057e-5c51-449d-8390-7879cae4f8c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:11.690230 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:11.690199 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5570057e-5c51-449d-8390-7879cae4f8c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:11.690464 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:11.690332 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 10:06:11.690464 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:11.690352 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 03 10:06:11.690464 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:11.690405 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls podName:c6b0923e-598e-4d07-bbd0-cdf55b61a8af nodeName:}" failed. No retries permitted until 2026-07-03 10:06:19.690384324 +0000 UTC m=+50.728530928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls") pod "dns-default-78277" (UID: "c6b0923e-598e-4d07-bbd0-cdf55b61a8af") : secret "dns-default-metrics-tls" not found Jul 03 10:06:11.690464 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:11.690421 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 10:06:11.690464 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:11.690424 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5570057e-5c51-449d-8390-7879cae4f8c8-kube-rbac-proxy-cm podName:5570057e-5c51-449d-8390-7879cae4f8c8 nodeName:}" failed. No retries permitted until 2026-07-03 10:06:13.690414209 +0000 UTC m=+44.728560800 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5570057e-5c51-449d-8390-7879cae4f8c8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6cgf4" (UID: "5570057e-5c51-449d-8390-7879cae4f8c8") : configmap "kube-rbac-proxy" not found Jul 03 10:06:11.690660 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:11.690479 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5570057e-5c51-449d-8390-7879cae4f8c8-insights-runtime-extractor-tls podName:5570057e-5c51-449d-8390-7879cae4f8c8 nodeName:}" failed. No retries permitted until 2026-07-03 10:06:13.690463695 +0000 UTC m=+44.728610283 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5570057e-5c51-449d-8390-7879cae4f8c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6cgf4" (UID: "5570057e-5c51-449d-8390-7879cae4f8c8") : secret "insights-runtime-extractor-tls" not found Jul 03 10:06:11.713565 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:11.713532 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-h7ks5" event={"ID":"f3f161fb-f33b-4aab-aa23-3a42315d24b0","Type":"ContainerStarted","Data":"3706a7a7410a338d89d23316b48e2446bc7f77d20f3938f3a8f265007f05307e"} Jul 03 10:06:12.536159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.536124 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-5fvvs"] Jul 03 10:06:12.553423 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.553400 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-5fvvs"] Jul 03 10:06:12.553547 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.553486 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-5fvvs" Jul 03 10:06:12.556251 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.556189 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 03 10:06:12.556551 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.556528 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 03 10:06:12.559658 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.559627 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 03 10:06:12.559807 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.559683 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ljxnn\"" Jul 03 10:06:12.559807 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.559729 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 03 10:06:12.573469 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.573439 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:12.596613 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.596582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a-signing-key\") pod \"service-ca-758f599c5-5fvvs\" (UID: \"c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a\") " pod="openshift-service-ca/service-ca-758f599c5-5fvvs" Jul 03 10:06:12.596613 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.596616 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hv46\" (UniqueName: \"kubernetes.io/projected/c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a-kube-api-access-9hv46\") pod \"service-ca-758f599c5-5fvvs\" (UID: \"c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a\") " pod="openshift-service-ca/service-ca-758f599c5-5fvvs" Jul 03 10:06:12.596789 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.596641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a-signing-cabundle\") pod \"service-ca-758f599c5-5fvvs\" (UID: \"c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a\") " pod="openshift-service-ca/service-ca-758f599c5-5fvvs" Jul 03 10:06:12.697206 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.697171 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a-signing-key\") pod \"service-ca-758f599c5-5fvvs\" (UID: \"c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a\") " pod="openshift-service-ca/service-ca-758f599c5-5fvvs" Jul 03 10:06:12.697206 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.697209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hv46\" (UniqueName: \"kubernetes.io/projected/c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a-kube-api-access-9hv46\") pod \"service-ca-758f599c5-5fvvs\" (UID: \"c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a\") " pod="openshift-service-ca/service-ca-758f599c5-5fvvs" Jul 03 10:06:12.697391 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.697259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a-signing-cabundle\") pod \"service-ca-758f599c5-5fvvs\" (UID: \"c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a\") " pod="openshift-service-ca/service-ca-758f599c5-5fvvs" Jul 03 10:06:12.697952 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.697933 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a-signing-cabundle\") pod \"service-ca-758f599c5-5fvvs\" (UID: \"c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a\") " pod="openshift-service-ca/service-ca-758f599c5-5fvvs" Jul 03 10:06:12.699554 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.699533 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a-signing-key\") pod \"service-ca-758f599c5-5fvvs\" (UID: \"c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a\") " pod="openshift-service-ca/service-ca-758f599c5-5fvvs" Jul 03 10:06:12.710949 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.710920 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hv46\" (UniqueName: \"kubernetes.io/projected/c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a-kube-api-access-9hv46\") pod \"service-ca-758f599c5-5fvvs\" (UID: \"c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a\") " pod="openshift-service-ca/service-ca-758f599c5-5fvvs" Jul 03 10:06:12.759984 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.759963 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vbnml_85cda35a-33a5-4f89-96f4-a2b3f8b585f7/node-ca/0.log" Jul 03 10:06:12.866524 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:12.866500 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-5fvvs" Jul 03 10:06:13.034411 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:13.034374 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-5fvvs"] Jul 03 10:06:13.038005 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:06:13.037982 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc32eb2e9_5298_4f12_abd9_ca5c5b4f3d1a.slice/crio-ede432e0e3a01e5f06d0e3bfd520b053cdd2297eeab0f07afb34bd9f3c4245d2 WatchSource:0}: Error finding container ede432e0e3a01e5f06d0e3bfd520b053cdd2297eeab0f07afb34bd9f3c4245d2: Status 404 returned error can't find the container with id ede432e0e3a01e5f06d0e3bfd520b053cdd2297eeab0f07afb34bd9f3c4245d2 Jul 03 10:06:13.301746 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:13.301707 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:13.361475 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:13.361450 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qz8nn_30ca9240-21bd-4d28-85d7-5ac0d2f438bb/serve-healthcheck-canary/0.log" Jul 03 10:06:13.468590 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:13.468564 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:13.705557 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:13.705518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5570057e-5c51-449d-8390-7879cae4f8c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:13.705714 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:13.705573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5570057e-5c51-449d-8390-7879cae4f8c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:13.705714 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:13.705674 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 10:06:13.705816 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:13.705776 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5570057e-5c51-449d-8390-7879cae4f8c8-insights-runtime-extractor-tls podName:5570057e-5c51-449d-8390-7879cae4f8c8 nodeName:}" failed. No retries permitted until 2026-07-03 10:06:17.705751612 +0000 UTC m=+48.743898227 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5570057e-5c51-449d-8390-7879cae4f8c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6cgf4" (UID: "5570057e-5c51-449d-8390-7879cae4f8c8") : secret "insights-runtime-extractor-tls" not found Jul 03 10:06:13.706258 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:13.706234 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5570057e-5c51-449d-8390-7879cae4f8c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:13.720245 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:13.720200 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-h7ks5" event={"ID":"f3f161fb-f33b-4aab-aa23-3a42315d24b0","Type":"ContainerStarted","Data":"43393d5bb397f1e42c02e10690ec2e8cf285684c88be10db343cca6a40f4d6f8"} Jul 03 10:06:13.720377 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:13.720275 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-h7ks5" event={"ID":"f3f161fb-f33b-4aab-aa23-3a42315d24b0","Type":"ContainerStarted","Data":"b9fed88193b75de2e9ca46f18ebfd5429e11e906ada19b95d230a1f10d51e4c1"} Jul 03 10:06:13.721408 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:13.721375 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-5fvvs" event={"ID":"c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a","Type":"ContainerStarted","Data":"ede432e0e3a01e5f06d0e3bfd520b053cdd2297eeab0f07afb34bd9f3c4245d2"} Jul 03 10:06:13.742428 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:13.742376 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-h7ks5" podStartSLOduration=1.948990196 podStartE2EDuration="3.742360637s" totalImagesPullingTime="1.793370441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:06:13.740855501 +0000 UTC m=+44.779002128" watchObservedRunningTime="2026-07-03 10:06:13.742360637 +0000 UTC m=+44.780507244" Jul 03 10:06:16.675017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:16.674926 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:16.731638 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:16.731590 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-5fvvs" event={"ID":"c32eb2e9-5298-4f12-abd9-ca5c5b4f3d1a","Type":"ContainerStarted","Data":"b407b321f016547fdc1fc9f312fc028b59995b1e570b1d5c2b37cf8e3687c696"} Jul 03 10:06:16.749552 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:16.749495 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-5fvvs" podStartSLOduration=2.097188332 podStartE2EDuration="4.74947382s" totalImagesPullingTime="2.652285488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:06:16.748150118 +0000 UTC m=+47.786296726" watchObservedRunningTime="2026-07-03 10:06:16.74947382 +0000 UTC m=+47.787620430" Jul 03 10:06:16.902579 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:16.902531 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:17.001284 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:17.001183 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:17.181812 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:17.181744 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:17.560740 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:17.560706 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:17.735306 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:17.735280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5570057e-5c51-449d-8390-7879cae4f8c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:17.735650 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:17.735407 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 10:06:17.735650 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:17.735469 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5570057e-5c51-449d-8390-7879cae4f8c8-insights-runtime-extractor-tls podName:5570057e-5c51-449d-8390-7879cae4f8c8 nodeName:}" failed. No retries permitted until 2026-07-03 10:06:25.735453511 +0000 UTC m=+56.773600095 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5570057e-5c51-449d-8390-7879cae4f8c8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6cgf4" (UID: "5570057e-5c51-449d-8390-7879cae4f8c8") : secret "insights-runtime-extractor-tls" not found Jul 03 10:06:17.979167 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:17.979104 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:18.055228 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:18.055152 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:18.773318 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:18.773275 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:18.864027 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:18.863999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:18.965855 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:18.965824 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:19.410512 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:19.410485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:19.570916 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:19.570888 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:19.652607 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:19.652570 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:19.652732 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:19.652712 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 03 10:06:19.652732 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:19.652730 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59748c45cf-vll4w: secret "image-registry-tls" not found Jul 03 10:06:19.652866 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:19.652783 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls podName:0eac25e7-d923-45c0-98ca-69884664d7fc nodeName:}" failed. No retries permitted until 2026-07-03 10:06:35.652767057 +0000 UTC m=+66.690913642 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls") pod "image-registry-59748c45cf-vll4w" (UID: "0eac25e7-d923-45c0-98ca-69884664d7fc") : secret "image-registry-tls" not found Jul 03 10:06:19.753429 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:19.753355 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls\") pod \"dns-default-78277\" (UID: \"c6b0923e-598e-4d07-bbd0-cdf55b61a8af\") " pod="openshift-dns/dns-default-78277" Jul 03 10:06:19.753562 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:19.753469 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 10:06:19.753562 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:06:19.753523 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls podName:c6b0923e-598e-4d07-bbd0-cdf55b61a8af nodeName:}" failed. No retries permitted until 2026-07-03 10:06:35.753506178 +0000 UTC m=+66.791652765 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls") pod "dns-default-78277" (UID: "c6b0923e-598e-4d07-bbd0-cdf55b61a8af") : secret "dns-default-metrics-tls" not found Jul 03 10:06:20.167782 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:20.167752 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:20.358212 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:20.358184 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:20.474551 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:20.474472 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:20.572330 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:20.572295 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:20.759835 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:20.759749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:20.962862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:20.962833 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:21.367187 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:21.367156 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:21.424860 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:21.424828 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:21.561519 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:21.561485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:22.170468 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:22.170437 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:22.592348 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:22.592260 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:22.978281 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:22.978253 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:23.047201 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:23.047153 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:23.261175 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:23.261092 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:23.363264 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:23.363204 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:23.453792 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:23.453759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:23.610186 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:23.610155 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:24.184230 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:24.184185 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:24.759814 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:24.759779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:24.968990 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:24.968957 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:25.167992 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:25.167954 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:25.563428 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:25.563343 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:25.802089 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:25.802056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5570057e-5c51-449d-8390-7879cae4f8c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:25.804179 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:25.804159 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5570057e-5c51-449d-8390-7879cae4f8c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6cgf4\" (UID: \"5570057e-5c51-449d-8390-7879cae4f8c8\") " pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:25.911932 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:25.911896 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6cgf4" Jul 03 10:06:26.031863 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:26.031826 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6cgf4"] Jul 03 10:06:26.035443 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:06:26.035411 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5570057e_5c51_449d_8390_7879cae4f8c8.slice/crio-3259aa34567993ff227b0fd29815b60fcc0b1078d5609f7fbfaa49b649224207 WatchSource:0}: Error finding container 3259aa34567993ff227b0fd29815b60fcc0b1078d5609f7fbfaa49b649224207: Status 404 returned error can't find the container with id 3259aa34567993ff227b0fd29815b60fcc0b1078d5609f7fbfaa49b649224207 Jul 03 10:06:26.360369 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:26.360340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:26.757399 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:26.757274 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:26.757979 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:26.757880 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6cgf4" event={"ID":"5570057e-5c51-449d-8390-7879cae4f8c8","Type":"ContainerStarted","Data":"3259aa34567993ff227b0fd29815b60fcc0b1078d5609f7fbfaa49b649224207"} Jul 03 10:06:26.963480 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:26.963419 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:27.159533 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:27.159502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:27.373493 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:27.373462 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:30.686041 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:30.686010 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-knz5k" Jul 03 10:06:30.769955 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:30.769919 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6cgf4" event={"ID":"5570057e-5c51-449d-8390-7879cae4f8c8","Type":"ContainerStarted","Data":"9f0d5fc7c1e714ca26d9cb7d482b6a518b3c701ac38d10e08c25401ad627e1bb"} Jul 03 10:06:31.073943 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:31.073860 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:31.175768 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:31.175736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:31.276855 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:31.276822 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:31.377759 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:31.377724 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:31.533234 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:31.533193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:31.604547 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:31.604511 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:31.760475 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:31.760391 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:31.775185 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:31.775154 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6cgf4" event={"ID":"5570057e-5c51-449d-8390-7879cae4f8c8","Type":"ContainerStarted","Data":"4cf41599774ae163a868083973c14300e7f63de5b5e1f2523960278366ba4d66"} Jul 03 10:06:31.821043 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:31.821000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:31.981300 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:31.981263 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:32.084059 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:32.083980 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:32.183604 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:32.183576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:32.744234 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:32.744180 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:32.938776 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:32.938732 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:33.131807 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:33.131763 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:33.783367 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:33.783279 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6cgf4" event={"ID":"5570057e-5c51-449d-8390-7879cae4f8c8","Type":"ContainerStarted","Data":"cdd37f63d29602517857866e6fc659fcf5d623bb07c28dde6bd7c3680968e7be"} Jul 03 10:06:33.805272 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:33.805206 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6cgf4" podStartSLOduration=17.658833317 podStartE2EDuration="24.805190206s" totalImagesPullingTime="7.146356889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 10:06:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:06:33.803960357 +0000 UTC m=+64.842106962" watchObservedRunningTime="2026-07-03 10:06:33.805190206 +0000 UTC m=+64.843336813" Jul 03 10:06:33.968342 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:33.968308 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:34.043047 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:34.042977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:35.180407 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.180355 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs\") pod \"network-metrics-daemon-lwkn9\" (UID: \"de5822f2-b52a-4283-938f-0395efeae942\") " pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:06:35.183144 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.183111 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 03 10:06:35.192757 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.192727 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5822f2-b52a-4283-938f-0395efeae942-metrics-certs\") pod \"network-metrics-daemon-lwkn9\" (UID: \"de5822f2-b52a-4283-938f-0395efeae942\") " pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:06:35.280905 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.280866 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret\") pod \"global-pull-secret-syncer-pzm5l\" (UID: \"a6273a00-72c1-456e-b47d-9a24d012f8c0\") " pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:06:35.283270 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.283253 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 03 10:06:35.293872 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.293842 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a6273a00-72c1-456e-b47d-9a24d012f8c0-original-pull-secret\") pod \"global-pull-secret-syncer-pzm5l\" (UID: \"a6273a00-72c1-456e-b47d-9a24d012f8c0\") " pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:06:35.381728 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.381690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbd6f\" (UniqueName: \"kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f\") pod \"network-check-target-pzhjd\" (UID: \"abbde693-f1f4-469f-bc67-c793cb2007de\") " pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:06:35.386414 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.386395 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 03 10:06:35.394896 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.394882 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 03 10:06:35.406004 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.405976 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbd6f\" (UniqueName: \"kubernetes.io/projected/abbde693-f1f4-469f-bc67-c793cb2007de-kube-api-access-vbd6f\") pod \"network-check-target-pzhjd\" (UID: \"abbde693-f1f4-469f-bc67-c793cb2007de\") " pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:06:35.439886 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.439803 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xh5d7\"" Jul 03 10:06:35.447354 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.447335 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lwkn9" Jul 03 10:06:35.447593 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.447577 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-njhrs\"" Jul 03 10:06:35.456484 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.456466 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:06:35.536929 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.536905 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pzm5l" Jul 03 10:06:35.575370 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.575336 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lwkn9"] Jul 03 10:06:35.578734 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:06:35.578706 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5822f2_b52a_4283_938f_0395efeae942.slice/crio-8fe38fc14240ea7e78c11c427dbfaf7d3f62de861470966635bfc4a442bd9c25 WatchSource:0}: Error finding container 8fe38fc14240ea7e78c11c427dbfaf7d3f62de861470966635bfc4a442bd9c25: Status 404 returned error can't find the container with id 8fe38fc14240ea7e78c11c427dbfaf7d3f62de861470966635bfc4a442bd9c25 Jul 03 10:06:35.596085 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.596059 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pzhjd"] Jul 03 10:06:35.668020 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.667988 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pzm5l"] Jul 03 10:06:35.671089 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:06:35.671066 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6273a00_72c1_456e_b47d_9a24d012f8c0.slice/crio-12aafe402d4029e72b6edcf5326c5094df2a2aaac61e6a4a2f66868df0c78df8 WatchSource:0}: Error finding container 12aafe402d4029e72b6edcf5326c5094df2a2aaac61e6a4a2f66868df0c78df8: Status 404 returned error can't find the container with id 12aafe402d4029e72b6edcf5326c5094df2a2aaac61e6a4a2f66868df0c78df8 Jul 03 10:06:35.683844 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.683823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:35.686078 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.686059 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls\") pod \"image-registry-59748c45cf-vll4w\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:35.785186 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.785100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls\") pod \"dns-default-78277\" (UID: \"c6b0923e-598e-4d07-bbd0-cdf55b61a8af\") " pod="openshift-dns/dns-default-78277" Jul 03 10:06:35.787409 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.787384 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6b0923e-598e-4d07-bbd0-cdf55b61a8af-metrics-tls\") pod \"dns-default-78277\" (UID: \"c6b0923e-598e-4d07-bbd0-cdf55b61a8af\") " pod="openshift-dns/dns-default-78277" Jul 03 10:06:35.790428 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.790396 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lwkn9" event={"ID":"de5822f2-b52a-4283-938f-0395efeae942","Type":"ContainerStarted","Data":"8fe38fc14240ea7e78c11c427dbfaf7d3f62de861470966635bfc4a442bd9c25"} Jul 03 10:06:35.791462 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.791440 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pzm5l" event={"ID":"a6273a00-72c1-456e-b47d-9a24d012f8c0","Type":"ContainerStarted","Data":"12aafe402d4029e72b6edcf5326c5094df2a2aaac61e6a4a2f66868df0c78df8"} Jul 03 10:06:35.792386 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.792370 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pzhjd" event={"ID":"abbde693-f1f4-469f-bc67-c793cb2007de","Type":"ContainerStarted","Data":"c3ac79604032960d03cb0327a631d195e5513a4b607af4e3f2e73dfb38797454"} Jul 03 10:06:35.947771 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.947744 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zmbwq\"" Jul 03 10:06:35.956348 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.956328 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:35.972832 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.972807 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sgbfl\"" Jul 03 10:06:35.981556 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:35.981534 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-78277" Jul 03 10:06:36.106090 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:36.105801 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59748c45cf-vll4w"] Jul 03 10:06:36.108993 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:06:36.108955 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eac25e7_d923_45c0_98ca_69884664d7fc.slice/crio-5a20ce03e1383b6ecb46f878a3d93bf0edeb00d62812dfc427b879ffc160d81d WatchSource:0}: Error finding container 5a20ce03e1383b6ecb46f878a3d93bf0edeb00d62812dfc427b879ffc160d81d: Status 404 returned error can't find the container with id 5a20ce03e1383b6ecb46f878a3d93bf0edeb00d62812dfc427b879ffc160d81d Jul 03 10:06:36.127705 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:36.127634 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-78277"] Jul 03 10:06:36.132463 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:06:36.132425 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b0923e_598e_4d07_bbd0_cdf55b61a8af.slice/crio-af60dd44d7b2f708378d5274eb020650b14d7c54400fc68e4855070bfda68401 WatchSource:0}: Error finding container af60dd44d7b2f708378d5274eb020650b14d7c54400fc68e4855070bfda68401: Status 404 returned error can't find the container with id af60dd44d7b2f708378d5274eb020650b14d7c54400fc68e4855070bfda68401 Jul 03 10:06:36.798365 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:36.798324 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59748c45cf-vll4w" event={"ID":"0eac25e7-d923-45c0-98ca-69884664d7fc","Type":"ContainerStarted","Data":"20e8f5ee47810193681884c07f9fef4c9bd72879fd310bdf448e3f9a6b4f36fb"} Jul 03 10:06:36.798808 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:36.798393 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:06:36.798808 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:36.798418 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59748c45cf-vll4w" event={"ID":"0eac25e7-d923-45c0-98ca-69884664d7fc","Type":"ContainerStarted","Data":"5a20ce03e1383b6ecb46f878a3d93bf0edeb00d62812dfc427b879ffc160d81d"} Jul 03 10:06:36.799658 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:36.799627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-78277" event={"ID":"c6b0923e-598e-4d07-bbd0-cdf55b61a8af","Type":"ContainerStarted","Data":"af60dd44d7b2f708378d5274eb020650b14d7c54400fc68e4855070bfda68401"} Jul 03 10:06:36.821952 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:36.821745 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59748c45cf-vll4w" podStartSLOduration=47.82172911 podStartE2EDuration="47.82172911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:06:36.820435204 +0000 UTC m=+67.858581820" watchObservedRunningTime="2026-07-03 10:06:36.82172911 +0000 UTC m=+67.859875717" Jul 03 10:06:37.195196 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.195163 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:37.296706 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.296650 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:37.395272 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.395181 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:37.488535 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.488409 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-9hhgb"] Jul 03 10:06:37.511395 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.511366 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68c647fd-f4k2s"] Jul 03 10:06:37.511914 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.511772 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-9hhgb" Jul 03 10:06:37.516041 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.515586 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-g2lsb\"" Jul 03 10:06:37.516041 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.515851 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 03 10:06:37.516976 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.516807 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 03 10:06:37.533701 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.533673 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68c647fd-f4k2s" Jul 03 10:06:37.537335 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.536939 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 03 10:06:37.537335 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.537164 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 03 10:06:37.541438 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.541021 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-9hhgb"] Jul 03 10:06:37.541438 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.541066 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68c647fd-f4k2s"] Jul 03 10:06:37.541438 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.541086 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59748c45cf-vll4w"] Jul 03 10:06:37.542990 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.542663 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 03 10:06:37.545147 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.543685 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf"] Jul 03 10:06:37.545147 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.543784 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 03 10:06:37.545147 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.544258 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-n2dkq\"" Jul 03 10:06:37.561998 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.561641 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.565301 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.564885 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 03 10:06:37.565301 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.565190 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 03 10:06:37.568627 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.568408 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 03 10:06:37.568733 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.568677 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 03 10:06:37.584425 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.584368 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf"] Jul 03 10:06:37.597988 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.597881 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:37.602033 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.601810 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0319f4ec-7dd2-4ca9-9f58-1ba5e6f33ca7-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-9hhgb\" (UID: \"0319f4ec-7dd2-4ca9-9f58-1ba5e6f33ca7\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-9hhgb" Jul 03 10:06:37.602033 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.601866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b5914559-cf45-467a-9126-843c67952d6d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f68c647fd-f4k2s\" (UID: \"b5914559-cf45-467a-9126-843c67952d6d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68c647fd-f4k2s" Jul 03 10:06:37.602033 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.601929 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0319f4ec-7dd2-4ca9-9f58-1ba5e6f33ca7-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-9hhgb\" (UID: \"0319f4ec-7dd2-4ca9-9f58-1ba5e6f33ca7\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-9hhgb" Jul 03 10:06:37.602033 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.601986 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpvzl\" (UniqueName: \"kubernetes.io/projected/b5914559-cf45-467a-9126-843c67952d6d-kube-api-access-vpvzl\") pod \"managed-serviceaccount-addon-agent-7f68c647fd-f4k2s\" (UID: \"b5914559-cf45-467a-9126-843c67952d6d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68c647fd-f4k2s" Jul 03 10:06:37.678174 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.678142 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cdf799895-6vjq7"] Jul 03 10:06:37.704190 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.703150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpvzl\" (UniqueName: \"kubernetes.io/projected/b5914559-cf45-467a-9126-843c67952d6d-kube-api-access-vpvzl\") pod \"managed-serviceaccount-addon-agent-7f68c647fd-f4k2s\" (UID: \"b5914559-cf45-467a-9126-843c67952d6d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68c647fd-f4k2s" Jul 03 10:06:37.704190 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.703195 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/411cccd8-5685-40c0-a79b-53706aaa8931-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-695fc9c645-95ltf\" (UID: \"411cccd8-5685-40c0-a79b-53706aaa8931\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.704190 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.703240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/411cccd8-5685-40c0-a79b-53706aaa8931-additional-service-ca\") pod \"cluster-proxy-proxy-agent-695fc9c645-95ltf\" (UID: \"411cccd8-5685-40c0-a79b-53706aaa8931\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.704190 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.703272 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/411cccd8-5685-40c0-a79b-53706aaa8931-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-695fc9c645-95ltf\" (UID: \"411cccd8-5685-40c0-a79b-53706aaa8931\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.704190 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.703306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/411cccd8-5685-40c0-a79b-53706aaa8931-ca\") pod \"cluster-proxy-proxy-agent-695fc9c645-95ltf\" (UID: \"411cccd8-5685-40c0-a79b-53706aaa8931\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.704190 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.703335 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0319f4ec-7dd2-4ca9-9f58-1ba5e6f33ca7-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-9hhgb\" (UID: \"0319f4ec-7dd2-4ca9-9f58-1ba5e6f33ca7\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-9hhgb" Jul 03 10:06:37.704190 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.703372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b5914559-cf45-467a-9126-843c67952d6d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f68c647fd-f4k2s\" (UID: \"b5914559-cf45-467a-9126-843c67952d6d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68c647fd-f4k2s" Jul 03 10:06:37.704190 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.703396 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/411cccd8-5685-40c0-a79b-53706aaa8931-hub\") pod \"cluster-proxy-proxy-agent-695fc9c645-95ltf\" (UID: \"411cccd8-5685-40c0-a79b-53706aaa8931\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.704190 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.703453 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp9qg\" (UniqueName: \"kubernetes.io/projected/411cccd8-5685-40c0-a79b-53706aaa8931-kube-api-access-qp9qg\") pod \"cluster-proxy-proxy-agent-695fc9c645-95ltf\" (UID: \"411cccd8-5685-40c0-a79b-53706aaa8931\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.704190 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.703488 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0319f4ec-7dd2-4ca9-9f58-1ba5e6f33ca7-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-9hhgb\" (UID: \"0319f4ec-7dd2-4ca9-9f58-1ba5e6f33ca7\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-9hhgb" Jul 03 10:06:37.704190 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.703746 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cdf799895-6vjq7"] Jul 03 10:06:37.704190 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.703862 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:37.704190 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.703883 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.704190 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.704096 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0319f4ec-7dd2-4ca9-9f58-1ba5e6f33ca7-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-9hhgb\" (UID: \"0319f4ec-7dd2-4ca9-9f58-1ba5e6f33ca7\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-9hhgb" Jul 03 10:06:37.708771 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.708722 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0319f4ec-7dd2-4ca9-9f58-1ba5e6f33ca7-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-9hhgb\" (UID: \"0319f4ec-7dd2-4ca9-9f58-1ba5e6f33ca7\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-9hhgb" Jul 03 10:06:37.713873 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.713826 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpvzl\" (UniqueName: \"kubernetes.io/projected/b5914559-cf45-467a-9126-843c67952d6d-kube-api-access-vpvzl\") pod \"managed-serviceaccount-addon-agent-7f68c647fd-f4k2s\" (UID: \"b5914559-cf45-467a-9126-843c67952d6d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68c647fd-f4k2s" Jul 03 10:06:37.714072 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.714051 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b5914559-cf45-467a-9126-843c67952d6d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7f68c647fd-f4k2s\" (UID: \"b5914559-cf45-467a-9126-843c67952d6d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68c647fd-f4k2s" Jul 03 10:06:37.804710 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.803848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-registry-tls\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.804710 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.803890 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-registry-certificates\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.804710 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.803933 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-installation-pull-secrets\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.804710 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.803979 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp9qg\" (UniqueName: \"kubernetes.io/projected/411cccd8-5685-40c0-a79b-53706aaa8931-kube-api-access-qp9qg\") pod \"cluster-proxy-proxy-agent-695fc9c645-95ltf\" (UID: \"411cccd8-5685-40c0-a79b-53706aaa8931\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.804710 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.804018 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-image-registry-private-configuration\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.804710 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.804045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/411cccd8-5685-40c0-a79b-53706aaa8931-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-695fc9c645-95ltf\" (UID: \"411cccd8-5685-40c0-a79b-53706aaa8931\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.804710 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.804072 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/411cccd8-5685-40c0-a79b-53706aaa8931-additional-service-ca\") pod \"cluster-proxy-proxy-agent-695fc9c645-95ltf\" (UID: \"411cccd8-5685-40c0-a79b-53706aaa8931\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.805486 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.804809 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/411cccd8-5685-40c0-a79b-53706aaa8931-additional-service-ca\") pod \"cluster-proxy-proxy-agent-695fc9c645-95ltf\" (UID: \"411cccd8-5685-40c0-a79b-53706aaa8931\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.805486 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.804886 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-bound-sa-token\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.805486 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.804928 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nmzm\" (UniqueName: \"kubernetes.io/projected/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-kube-api-access-8nmzm\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.805486 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.804968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/411cccd8-5685-40c0-a79b-53706aaa8931-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-695fc9c645-95ltf\" (UID: \"411cccd8-5685-40c0-a79b-53706aaa8931\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.805486 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.805465 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/411cccd8-5685-40c0-a79b-53706aaa8931-ca\") pod \"cluster-proxy-proxy-agent-695fc9c645-95ltf\" (UID: \"411cccd8-5685-40c0-a79b-53706aaa8931\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.805717 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.805496 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-ca-trust-extracted\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.805717 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.805537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/411cccd8-5685-40c0-a79b-53706aaa8931-hub\") pod \"cluster-proxy-proxy-agent-695fc9c645-95ltf\" (UID: \"411cccd8-5685-40c0-a79b-53706aaa8931\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.805717 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.805564 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-trusted-ca\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.808166 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.808106 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/411cccd8-5685-40c0-a79b-53706aaa8931-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-695fc9c645-95ltf\" (UID: \"411cccd8-5685-40c0-a79b-53706aaa8931\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.810144 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.810102 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/411cccd8-5685-40c0-a79b-53706aaa8931-ca\") pod \"cluster-proxy-proxy-agent-695fc9c645-95ltf\" (UID: \"411cccd8-5685-40c0-a79b-53706aaa8931\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.810509 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.810486 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/411cccd8-5685-40c0-a79b-53706aaa8931-hub\") pod \"cluster-proxy-proxy-agent-695fc9c645-95ltf\" (UID: \"411cccd8-5685-40c0-a79b-53706aaa8931\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.811664 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.811603 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/411cccd8-5685-40c0-a79b-53706aaa8931-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-695fc9c645-95ltf\" (UID: \"411cccd8-5685-40c0-a79b-53706aaa8931\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.811768 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.811686 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lwkn9" event={"ID":"de5822f2-b52a-4283-938f-0395efeae942","Type":"ContainerStarted","Data":"3d5934ad1d9fade23a1fbb24e3ae3e0fb76ae278ae088f2b0e7ea6c22e18f9ca"} Jul 03 10:06:37.811768 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.811730 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lwkn9" event={"ID":"de5822f2-b52a-4283-938f-0395efeae942","Type":"ContainerStarted","Data":"78c731023990a4554452ebfd4fca9cf72f962d95aa81422b2c884f3e69b9fc69"} Jul 03 10:06:37.822568 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.822520 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp9qg\" (UniqueName: \"kubernetes.io/projected/411cccd8-5685-40c0-a79b-53706aaa8931-kube-api-access-qp9qg\") pod \"cluster-proxy-proxy-agent-695fc9c645-95ltf\" (UID: \"411cccd8-5685-40c0-a79b-53706aaa8931\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.826139 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.826116 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-9hhgb" Jul 03 10:06:37.833950 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.833888 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lwkn9" podStartSLOduration=67.251447447 podStartE2EDuration="1m8.833868836s" totalImagesPullingTime="1.582421389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:05:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:06:37.832774395 +0000 UTC m=+68.870921004" watchObservedRunningTime="2026-07-03 10:06:37.833868836 +0000 UTC m=+68.872015444" Jul 03 10:06:37.850121 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.850087 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:37.854558 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.854525 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68c647fd-f4k2s" Jul 03 10:06:37.885386 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.885173 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" Jul 03 10:06:37.906643 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.906067 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-trusted-ca\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.906643 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.906143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-registry-tls\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.906643 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.906174 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-registry-certificates\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.906643 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.906211 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-installation-pull-secrets\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.906643 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.906431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-image-registry-private-configuration\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.906643 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.906495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-bound-sa-token\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.906643 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.906522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nmzm\" (UniqueName: \"kubernetes.io/projected/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-kube-api-access-8nmzm\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.906643 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.906567 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-ca-trust-extracted\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.907135 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.906990 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-ca-trust-extracted\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.907848 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.907745 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-registry-certificates\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.909005 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.908981 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-trusted-ca\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.910303 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.910237 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-image-registry-private-configuration\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.912683 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.912368 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-installation-pull-secrets\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.913226 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.913190 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-registry-tls\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.928097 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.928021 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-bound-sa-token\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.941245 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.940798 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nmzm\" (UniqueName: \"kubernetes.io/projected/5f586edf-f0d1-45f9-9f55-330eed0dc1a4-kube-api-access-8nmzm\") pod \"image-registry-cdf799895-6vjq7\" (UID: \"5f586edf-f0d1-45f9-9f55-330eed0dc1a4\") " pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:37.976770 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:37.976689 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:38.042577 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:38.042253 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:38.048907 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:38.048868 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-9hhgb"] Jul 03 10:06:38.052533 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:06:38.052500 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0319f4ec_7dd2_4ca9_9f58_1ba5e6f33ca7.slice/crio-3bd73a0a3ce9e184009d3cefb7509dca363827ad9a0178bd82ffe4e1344b7d16 WatchSource:0}: Error finding container 3bd73a0a3ce9e184009d3cefb7509dca363827ad9a0178bd82ffe4e1344b7d16: Status 404 returned error can't find the container with id 3bd73a0a3ce9e184009d3cefb7509dca363827ad9a0178bd82ffe4e1344b7d16 Jul 03 10:06:38.073774 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:38.073715 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68c647fd-f4k2s"] Jul 03 10:06:38.086994 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:38.086969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:38.134441 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:38.134411 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf"] Jul 03 10:06:38.240081 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:38.240042 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cdf799895-6vjq7"] Jul 03 10:06:38.317050 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:38.317001 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:38.391491 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:38.391458 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:38.552687 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:38.552586 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:38.816484 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:38.816392 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68c647fd-f4k2s" event={"ID":"b5914559-cf45-467a-9126-843c67952d6d","Type":"ContainerStarted","Data":"20cea9c6d03c3d3429405d4848c75b335a8c6ab907de70f134b05f64fd21746e"} Jul 03 10:06:38.817773 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:38.817747 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-9hhgb" event={"ID":"0319f4ec-7dd2-4ca9-9f58-1ba5e6f33ca7","Type":"ContainerStarted","Data":"3bd73a0a3ce9e184009d3cefb7509dca363827ad9a0178bd82ffe4e1344b7d16"} Jul 03 10:06:38.990442 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:38.990403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:39.069367 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:39.069285 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:39.631744 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:06:39.631711 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod411cccd8_5685_40c0_a79b_53706aaa8931.slice/crio-af99a37840ccc229fc85e6f3ca8df0acfaf7dd5fd3279ab1a830dd9e299b28f5 WatchSource:0}: Error finding container af99a37840ccc229fc85e6f3ca8df0acfaf7dd5fd3279ab1a830dd9e299b28f5: Status 404 returned error can't find the container with id af99a37840ccc229fc85e6f3ca8df0acfaf7dd5fd3279ab1a830dd9e299b28f5 Jul 03 10:06:39.632646 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:06:39.632414 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f586edf_f0d1_45f9_9f55_330eed0dc1a4.slice/crio-16504ca6fd8f43f7aa8d512f67dad694f0a7644c03364531ea6668b7f2efade3 WatchSource:0}: Error finding container 16504ca6fd8f43f7aa8d512f67dad694f0a7644c03364531ea6668b7f2efade3: Status 404 returned error can't find the container with id 16504ca6fd8f43f7aa8d512f67dad694f0a7644c03364531ea6668b7f2efade3 Jul 03 10:06:39.822925 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:39.822892 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cdf799895-6vjq7" event={"ID":"5f586edf-f0d1-45f9-9f55-330eed0dc1a4","Type":"ContainerStarted","Data":"16504ca6fd8f43f7aa8d512f67dad694f0a7644c03364531ea6668b7f2efade3"} Jul 03 10:06:39.824373 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:39.824324 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" event={"ID":"411cccd8-5685-40c0-a79b-53706aaa8931","Type":"ContainerStarted","Data":"af99a37840ccc229fc85e6f3ca8df0acfaf7dd5fd3279ab1a830dd9e299b28f5"} Jul 03 10:06:40.394102 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:40.394059 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:40.493190 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:40.493154 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:40.753885 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:40.753805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:41.836847 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:41.836810 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-78277" event={"ID":"c6b0923e-598e-4d07-bbd0-cdf55b61a8af","Type":"ContainerStarted","Data":"0df0e785fbc19e6136d53b90feb0680ebb142995eb44e4df8a7b02fb366a4d24"} Jul 03 10:06:41.837433 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:41.836882 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-78277" event={"ID":"c6b0923e-598e-4d07-bbd0-cdf55b61a8af","Type":"ContainerStarted","Data":"1ef56820293dbeb94e24a78f5f33f205e1697fd48970de8a3c4888afd122feed"} Jul 03 10:06:41.837433 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:41.837097 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-78277" Jul 03 10:06:41.845018 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:41.844511 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pzm5l" event={"ID":"a6273a00-72c1-456e-b47d-9a24d012f8c0","Type":"ContainerStarted","Data":"02ea450d482fca5d8b124deb02dc9324be849e27e179c251923959172dd6fef0"} Jul 03 10:06:41.848302 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:41.847718 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cdf799895-6vjq7" event={"ID":"5f586edf-f0d1-45f9-9f55-330eed0dc1a4","Type":"ContainerStarted","Data":"11bc80f3ebcbd97482fe5b6a2a2255969fef56f78e99fd8d5b66c64e34ef185f"} Jul 03 10:06:41.848302 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:41.848261 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:06:41.851745 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:41.851632 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pzhjd" event={"ID":"abbde693-f1f4-469f-bc67-c793cb2007de","Type":"ContainerStarted","Data":"7185a599f2ac5bc0f63e59dda3c801202708d44c40c97bb61ad98584819da675"} Jul 03 10:06:41.852099 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:41.852057 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:06:41.856526 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:41.856368 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-78277" podStartSLOduration=33.86814509 podStartE2EDuration="38.85634908s" totalImagesPullingTime="4.98820399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:06:41.855208109 +0000 UTC m=+72.893354732" watchObservedRunningTime="2026-07-03 10:06:41.85634908 +0000 UTC m=+72.894495666" Jul 03 10:06:41.898426 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:41.898366 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-cdf799895-6vjq7" podStartSLOduration=4.8983432879999995 podStartE2EDuration="4.898343288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:06:41.895992908 +0000 UTC m=+72.934139532" watchObservedRunningTime="2026-07-03 10:06:41.898343288 +0000 UTC m=+72.936489879" Jul 03 10:06:41.925718 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:41.925629 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pzm5l" podStartSLOduration=66.461205303 podStartE2EDuration="1m11.925557581s" totalImagesPullingTime="5.464352278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 10:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:06:41.910823929 +0000 UTC m=+72.948970537" watchObservedRunningTime="2026-07-03 10:06:41.925557581 +0000 UTC m=+72.963704190" Jul 03 10:06:41.926561 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:41.926511 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pzhjd" podStartSLOduration=66.398170203 podStartE2EDuration="1m11.926499371s" totalImagesPullingTime="5.528329168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:06:41.925371165 +0000 UTC m=+72.963517773" watchObservedRunningTime="2026-07-03 10:06:41.926499371 +0000 UTC m=+72.964645979" Jul 03 10:06:42.862587 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:42.862545 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-9hhgb" event={"ID":"0319f4ec-7dd2-4ca9-9f58-1ba5e6f33ca7","Type":"ContainerStarted","Data":"c2ddce629b3187ee4cddd9ee53d8dc658ae7b81e8df807b3b9d067176830fd62"} Jul 03 10:06:42.878169 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:42.878110 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-9hhgb" podStartSLOduration=1.645857487 podStartE2EDuration="5.877766958s" totalImagesPullingTime="4.231909471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:06:42.877078632 +0000 UTC m=+73.915225240" watchObservedRunningTime="2026-07-03 10:06:42.877766958 +0000 UTC m=+73.915913568" Jul 03 10:06:43.035508 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:43.035474 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:43.101916 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:43.101877 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:44.042016 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:44.041981 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:44.111788 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:44.111753 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:44.870097 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:44.870063 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" event={"ID":"411cccd8-5685-40c0-a79b-53706aaa8931","Type":"ContainerStarted","Data":"2524a4d7a0c060988673c0f26cd57faf0f91578cd48a7510a4028ac7055a7464"} Jul 03 10:06:44.870206 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:44.870107 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" event={"ID":"411cccd8-5685-40c0-a79b-53706aaa8931","Type":"ContainerStarted","Data":"df95f02b5a3425f6e7d6973420c230b307c9e79dfe6c7eaf14269669aa87c7fb"} Jul 03 10:06:44.871352 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:44.871325 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68c647fd-f4k2s" event={"ID":"b5914559-cf45-467a-9126-843c67952d6d","Type":"ContainerStarted","Data":"bf7de9c09e84107bec32f5b968561c29b149d522a0f7a270ef1be87952670c87"} Jul 03 10:06:44.887037 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:44.886998 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7f68c647fd-f4k2s" podStartSLOduration=1.278696859 podStartE2EDuration="7.886983138s" totalImagesPullingTime="6.608286279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:06:44.886082566 +0000 UTC m=+75.924229197" watchObservedRunningTime="2026-07-03 10:06:44.886983138 +0000 UTC m=+75.925129749" Jul 03 10:06:44.991343 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:44.991267 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:45.092127 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:45.092094 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:45.346615 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:45.346590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:45.876282 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:45.876248 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" event={"ID":"411cccd8-5685-40c0-a79b-53706aaa8931","Type":"ContainerStarted","Data":"4e303a693db9200d62524029d59cfefe1add7475cf9e13f3d30ee0d7bc4bbae2"} Jul 03 10:06:47.664416 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:47.664378 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:48.005832 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.005750 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:48.075880 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.075822 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-695fc9c645-95ltf" podStartSLOduration=6.002629664 podStartE2EDuration="11.07579891s" totalImagesPullingTime="5.073169246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:06:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:06:45.907595877 +0000 UTC m=+76.945742484" watchObservedRunningTime="2026-07-03 10:06:48.07579891 +0000 UTC m=+79.113945530" Jul 03 10:06:48.076326 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.076305 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cxfgx"] Jul 03 10:06:48.081309 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.081288 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.084032 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.084011 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 03 10:06:48.084257 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.084244 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 03 10:06:48.084839 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.084820 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xbqqd\"" Jul 03 10:06:48.084947 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.084865 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 03 10:06:48.084947 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.084894 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 03 10:06:48.085981 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.085966 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 03 10:06:48.086047 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.086008 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 03 10:06:48.150002 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.149969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:48.190104 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.190070 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eaaba771-633f-4753-a355-8c7e314a2b1a-root\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.190277 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.190111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eaaba771-633f-4753-a355-8c7e314a2b1a-node-exporter-textfile\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.190277 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.190128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eaaba771-633f-4753-a355-8c7e314a2b1a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.190277 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.190154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eaaba771-633f-4753-a355-8c7e314a2b1a-node-exporter-accelerators-collector-config\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.190277 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.190234 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eaaba771-633f-4753-a355-8c7e314a2b1a-node-exporter-wtmp\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.190277 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.190274 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eaaba771-633f-4753-a355-8c7e314a2b1a-node-exporter-tls\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.190431 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.190332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eaaba771-633f-4753-a355-8c7e314a2b1a-metrics-client-ca\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.190431 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.190355 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eaaba771-633f-4753-a355-8c7e314a2b1a-sys\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.190431 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.190397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncvsc\" (UniqueName: \"kubernetes.io/projected/eaaba771-633f-4753-a355-8c7e314a2b1a-kube-api-access-ncvsc\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.246074 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.246046 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:48.291161 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.291084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eaaba771-633f-4753-a355-8c7e314a2b1a-metrics-client-ca\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.291161 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.291117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eaaba771-633f-4753-a355-8c7e314a2b1a-sys\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.291161 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.291152 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncvsc\" (UniqueName: \"kubernetes.io/projected/eaaba771-633f-4753-a355-8c7e314a2b1a-kube-api-access-ncvsc\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.291405 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.291210 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eaaba771-633f-4753-a355-8c7e314a2b1a-sys\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.291405 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.291258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eaaba771-633f-4753-a355-8c7e314a2b1a-root\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.291405 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.291291 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eaaba771-633f-4753-a355-8c7e314a2b1a-node-exporter-textfile\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.291405 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.291309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eaaba771-633f-4753-a355-8c7e314a2b1a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.291405 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.291333 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eaaba771-633f-4753-a355-8c7e314a2b1a-node-exporter-accelerators-collector-config\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.291405 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.291362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eaaba771-633f-4753-a355-8c7e314a2b1a-node-exporter-wtmp\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.291405 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.291369 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eaaba771-633f-4753-a355-8c7e314a2b1a-root\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.291405 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.291394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eaaba771-633f-4753-a355-8c7e314a2b1a-node-exporter-tls\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.291701 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.291506 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eaaba771-633f-4753-a355-8c7e314a2b1a-node-exporter-wtmp\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.291701 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.291624 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eaaba771-633f-4753-a355-8c7e314a2b1a-node-exporter-textfile\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.291780 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.291758 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eaaba771-633f-4753-a355-8c7e314a2b1a-metrics-client-ca\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.291837 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.291818 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eaaba771-633f-4753-a355-8c7e314a2b1a-node-exporter-accelerators-collector-config\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.293720 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.293695 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eaaba771-633f-4753-a355-8c7e314a2b1a-node-exporter-tls\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.293813 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.293778 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eaaba771-633f-4753-a355-8c7e314a2b1a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.352863 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.352827 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncvsc\" (UniqueName: \"kubernetes.io/projected/eaaba771-633f-4753-a355-8c7e314a2b1a-kube-api-access-ncvsc\") pod \"node-exporter-cxfgx\" (UID: \"eaaba771-633f-4753-a355-8c7e314a2b1a\") " pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.390815 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.390784 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cxfgx" Jul 03 10:06:48.402256 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:06:48.402229 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaaba771_633f_4753_a355_8c7e314a2b1a.slice/crio-89b9249fb877e332791a2eeac7159d5cb9b8fa4574a8c30f005ed30994e1f96f WatchSource:0}: Error finding container 89b9249fb877e332791a2eeac7159d5cb9b8fa4574a8c30f005ed30994e1f96f: Status 404 returned error can't find the container with id 89b9249fb877e332791a2eeac7159d5cb9b8fa4574a8c30f005ed30994e1f96f Jul 03 10:06:48.471498 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.471440 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:48.619294 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.619261 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:48.888640 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.888542 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:48.890026 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.889994 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxfgx" event={"ID":"eaaba771-633f-4753-a355-8c7e314a2b1a","Type":"ContainerStarted","Data":"89b9249fb877e332791a2eeac7159d5cb9b8fa4574a8c30f005ed30994e1f96f"} Jul 03 10:06:48.989480 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:48.989439 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:49.091320 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:49.091291 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:49.311978 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:49.311871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:49.894874 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:49.894844 2579 generic.go:356] "Generic (PLEG): container finished" podID="eaaba771-633f-4753-a355-8c7e314a2b1a" containerID="14d1325d6437b0dddffeae50ace7cc639319d39ab5bcc24f4a81348b1a5217e0" exitCode=0 Jul 03 10:06:49.895306 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:49.894929 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxfgx" event={"ID":"eaaba771-633f-4753-a355-8c7e314a2b1a","Type":"ContainerDied","Data":"14d1325d6437b0dddffeae50ace7cc639319d39ab5bcc24f4a81348b1a5217e0"} Jul 03 10:06:50.901829 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:50.901789 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxfgx" event={"ID":"eaaba771-633f-4753-a355-8c7e314a2b1a","Type":"ContainerStarted","Data":"fa7cff268a1fdc33f4cf7a3fc0def1bb17edef29170ae7b5bac6853556bcd2bb"} Jul 03 10:06:50.902192 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:50.901894 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxfgx" event={"ID":"eaaba771-633f-4753-a355-8c7e314a2b1a","Type":"ContainerStarted","Data":"1ab0b86cc5b0c1561e002f786848d580a1528e9979084d4a0d312545f4a1a35c"} Jul 03 10:06:50.946737 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:50.946701 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:51.036207 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:51.036105 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:51.255049 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:51.254963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:51.352685 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:51.352657 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:51.865352 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:51.865318 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-78277" Jul 03 10:06:51.885266 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:51.885200 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cxfgx" podStartSLOduration=2.7586689140000002 podStartE2EDuration="3.885182486s" totalImagesPullingTime="1.062155943s" totalInitContainerRuntime="64.357629ms" isStatefulPod=true podCreationTimestamp="2026-07-03 10:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:06:50.93190632 +0000 UTC m=+81.970052927" watchObservedRunningTime="2026-07-03 10:06:51.885182486 +0000 UTC m=+82.923329092" Jul 03 10:06:52.032575 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:52.032546 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:52.133442 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:52.133362 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:52.666824 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:52.666789 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:52.783647 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:52.783617 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:52.933988 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:52.933915 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:53.364498 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:53.364470 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:53.504102 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:53.504070 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:54.240614 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:54.240579 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:57.078573 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:57.078538 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:58.361632 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:58.361598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:58.443305 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:58.443274 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:06:58.823787 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:06:58.823714 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:07:00.267361 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:00.267325 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:00.364189 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:00.364160 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:00.500235 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:00.500185 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:00.602982 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:00.602952 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:00.703149 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:00.703119 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:01.633273 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:01.633241 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:01.733364 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:01.733332 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:02.207957 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:02.207927 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:02.303276 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:02.303245 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:03.484809 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:03.484775 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:03.663016 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:03.662977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:03.817876 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:03.817796 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:03.839717 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:03.839682 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-59748c45cf-vll4w" podUID="0eac25e7-d923-45c0-98ca-69884664d7fc" containerName="registry" containerID="cri-o://20e8f5ee47810193681884c07f9fef4c9bd72879fd310bdf448e3f9a6b4f36fb" gracePeriod=30 Jul 03 10:07:03.870574 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:03.870554 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cdf799895-6vjq7" Jul 03 10:07:03.952823 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:03.952797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:04.289206 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.289172 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:04.452729 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.452704 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:04.571446 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.571424 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:07:04.703712 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.703674 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:04.718586 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.718557 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0eac25e7-d923-45c0-98ca-69884664d7fc-trusted-ca\") pod \"0eac25e7-d923-45c0-98ca-69884664d7fc\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " Jul 03 10:07:04.718685 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.718598 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-bound-sa-token\") pod \"0eac25e7-d923-45c0-98ca-69884664d7fc\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " Jul 03 10:07:04.718685 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.718638 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0eac25e7-d923-45c0-98ca-69884664d7fc-image-registry-private-configuration\") pod \"0eac25e7-d923-45c0-98ca-69884664d7fc\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " Jul 03 10:07:04.718685 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.718654 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szpmp\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-kube-api-access-szpmp\") pod \"0eac25e7-d923-45c0-98ca-69884664d7fc\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " Jul 03 10:07:04.718685 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.718677 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0eac25e7-d923-45c0-98ca-69884664d7fc-installation-pull-secrets\") pod \"0eac25e7-d923-45c0-98ca-69884664d7fc\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " Jul 03 10:07:04.718875 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.718740 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-certificates\") pod \"0eac25e7-d923-45c0-98ca-69884664d7fc\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " Jul 03 10:07:04.718875 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.718809 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls\") pod \"0eac25e7-d923-45c0-98ca-69884664d7fc\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " Jul 03 10:07:04.718875 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.718833 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0eac25e7-d923-45c0-98ca-69884664d7fc-ca-trust-extracted\") pod \"0eac25e7-d923-45c0-98ca-69884664d7fc\" (UID: \"0eac25e7-d923-45c0-98ca-69884664d7fc\") " Jul 03 10:07:04.719291 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.719239 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eac25e7-d923-45c0-98ca-69884664d7fc-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0eac25e7-d923-45c0-98ca-69884664d7fc" (UID: "0eac25e7-d923-45c0-98ca-69884664d7fc"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 10:07:04.719790 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.719758 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0eac25e7-d923-45c0-98ca-69884664d7fc" (UID: "0eac25e7-d923-45c0-98ca-69884664d7fc"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 10:07:04.721402 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.721364 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eac25e7-d923-45c0-98ca-69884664d7fc-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0eac25e7-d923-45c0-98ca-69884664d7fc" (UID: "0eac25e7-d923-45c0-98ca-69884664d7fc"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 10:07:04.721512 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.721428 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-kube-api-access-szpmp" (OuterVolumeSpecName: "kube-api-access-szpmp") pod "0eac25e7-d923-45c0-98ca-69884664d7fc" (UID: "0eac25e7-d923-45c0-98ca-69884664d7fc"). InnerVolumeSpecName "kube-api-access-szpmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 10:07:04.721512 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.721448 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eac25e7-d923-45c0-98ca-69884664d7fc-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0eac25e7-d923-45c0-98ca-69884664d7fc" (UID: "0eac25e7-d923-45c0-98ca-69884664d7fc"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 10:07:04.721512 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.721500 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0eac25e7-d923-45c0-98ca-69884664d7fc" (UID: "0eac25e7-d923-45c0-98ca-69884664d7fc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 10:07:04.721679 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.721542 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0eac25e7-d923-45c0-98ca-69884664d7fc" (UID: "0eac25e7-d923-45c0-98ca-69884664d7fc"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 10:07:04.727494 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.727471 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eac25e7-d923-45c0-98ca-69884664d7fc-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0eac25e7-d923-45c0-98ca-69884664d7fc" (UID: "0eac25e7-d923-45c0-98ca-69884664d7fc"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:07:04.820118 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.820040 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-tls\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:07:04.820118 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.820073 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0eac25e7-d923-45c0-98ca-69884664d7fc-ca-trust-extracted\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:07:04.820118 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.820086 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0eac25e7-d923-45c0-98ca-69884664d7fc-trusted-ca\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:07:04.820118 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.820095 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-bound-sa-token\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:07:04.820118 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.820104 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0eac25e7-d923-45c0-98ca-69884664d7fc-image-registry-private-configuration\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:07:04.820118 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.820112 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szpmp\" (UniqueName: \"kubernetes.io/projected/0eac25e7-d923-45c0-98ca-69884664d7fc-kube-api-access-szpmp\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:07:04.820118 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.820120 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0eac25e7-d923-45c0-98ca-69884664d7fc-installation-pull-secrets\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:07:04.820449 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.820128 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0eac25e7-d923-45c0-98ca-69884664d7fc-registry-certificates\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:07:04.897098 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.897058 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:04.956031 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.956000 2579 generic.go:356] "Generic (PLEG): container finished" podID="0eac25e7-d923-45c0-98ca-69884664d7fc" containerID="20e8f5ee47810193681884c07f9fef4c9bd72879fd310bdf448e3f9a6b4f36fb" exitCode=0 Jul 03 10:07:04.956031 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.956039 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59748c45cf-vll4w" Jul 03 10:07:04.956263 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.956056 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59748c45cf-vll4w" event={"ID":"0eac25e7-d923-45c0-98ca-69884664d7fc","Type":"ContainerDied","Data":"20e8f5ee47810193681884c07f9fef4c9bd72879fd310bdf448e3f9a6b4f36fb"} Jul 03 10:07:04.956263 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.956100 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59748c45cf-vll4w" event={"ID":"0eac25e7-d923-45c0-98ca-69884664d7fc","Type":"ContainerDied","Data":"5a20ce03e1383b6ecb46f878a3d93bf0edeb00d62812dfc427b879ffc160d81d"} Jul 03 10:07:04.956263 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.956133 2579 scope.go:117] "RemoveContainer" containerID="20e8f5ee47810193681884c07f9fef4c9bd72879fd310bdf448e3f9a6b4f36fb" Jul 03 10:07:04.965822 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.965795 2579 scope.go:117] "RemoveContainer" containerID="20e8f5ee47810193681884c07f9fef4c9bd72879fd310bdf448e3f9a6b4f36fb" Jul 03 10:07:04.966141 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:07:04.966116 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e8f5ee47810193681884c07f9fef4c9bd72879fd310bdf448e3f9a6b4f36fb\": container with ID starting with 20e8f5ee47810193681884c07f9fef4c9bd72879fd310bdf448e3f9a6b4f36fb not found: ID does not exist" containerID="20e8f5ee47810193681884c07f9fef4c9bd72879fd310bdf448e3f9a6b4f36fb" Jul 03 10:07:04.966234 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:04.966151 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e8f5ee47810193681884c07f9fef4c9bd72879fd310bdf448e3f9a6b4f36fb"} err="failed to get container status \"20e8f5ee47810193681884c07f9fef4c9bd72879fd310bdf448e3f9a6b4f36fb\": rpc error: code = NotFound desc = could not find container \"20e8f5ee47810193681884c07f9fef4c9bd72879fd310bdf448e3f9a6b4f36fb\": container with ID starting with 20e8f5ee47810193681884c07f9fef4c9bd72879fd310bdf448e3f9a6b4f36fb not found: ID does not exist" Jul 03 10:07:05.027786 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:05.027754 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59748c45cf-vll4w"] Jul 03 10:07:05.028746 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:05.028725 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59748c45cf-vll4w"] Jul 03 10:07:05.529303 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:05.529275 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eac25e7-d923-45c0-98ca-69884664d7fc" path="/var/lib/kubelet/pods/0eac25e7-d923-45c0-98ca-69884664d7fc/volumes" Jul 03 10:07:06.105980 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:06.105941 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:06.211047 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:06.211010 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:09.369393 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:09.369355 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:09.435920 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:09.435888 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:13.868867 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:13.868839 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pzhjd" Jul 03 10:07:15.199387 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:15.199357 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:15.263649 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:15.263612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:22.808412 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:22.808374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:23.336397 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:23.336361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:23.435845 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:23.435816 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:33.895920 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:33.895860 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:07:53.466867 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:07:53.466831 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:08:07.261704 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:08:07.261657 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:08:07.394421 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:08:07.394387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:08:16.248750 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:08:16.248715 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:08:16.351991 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:08:16.351955 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:08:16.647489 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:08:16.647457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:08:16.956775 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:08:16.956691 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:08:17.026424 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:08:17.026387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:08:27.589570 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:08:27.589469 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:08:27.689213 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:08:27.689172 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:08:27.790771 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:08:27.790736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:09:10.339046 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:09:10.339002 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:09:10.437365 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:09:10.437330 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:09:22.518846 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:09:22.518798 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:09:22.619039 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:09:22.619010 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:09:25.269331 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:09:25.269297 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:09:25.349949 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:09:25.349917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:09:25.414751 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:09:25.414715 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:09:25.989520 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:09:25.989478 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:09:26.064416 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:09:26.064386 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:09:34.510840 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:09:34.510812 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:09:34.612577 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:09:34.612545 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:09:35.978635 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:09:35.978604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:09:36.081951 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:09:36.081917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:09:36.148643 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:09:36.148608 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:09:38.331004 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:09:38.330968 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:09:38.428949 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:09:38.428912 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:10:02.821046 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:02.820962 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:10:17.195822 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:17.195789 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:10:17.305033 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:17.305003 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:10:29.353393 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:29.353362 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 03 10:10:44.091866 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:44.091827 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:10:46.065058 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.065019 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4nknq"] Jul 03 10:10:46.065430 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.065262 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eac25e7-d923-45c0-98ca-69884664d7fc" containerName="registry" Jul 03 10:10:46.065430 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.065273 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eac25e7-d923-45c0-98ca-69884664d7fc" containerName="registry" Jul 03 10:10:46.065430 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.065325 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eac25e7-d923-45c0-98ca-69884664d7fc" containerName="registry" Jul 03 10:10:46.067005 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.066990 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4nknq" Jul 03 10:10:46.069252 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.069234 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-r9j2k\"" Jul 03 10:10:46.069488 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.069475 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 03 10:10:46.070484 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.070467 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 03 10:10:46.083411 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.083385 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4nknq"] Jul 03 10:10:46.085350 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.085325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd6faf27-4405-43fe-a7e8-6ee3b1055c69-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-4nknq\" (UID: \"cd6faf27-4405-43fe-a7e8-6ee3b1055c69\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4nknq" Jul 03 10:10:46.085469 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.085378 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqxjm\" (UniqueName: \"kubernetes.io/projected/cd6faf27-4405-43fe-a7e8-6ee3b1055c69-kube-api-access-pqxjm\") pod \"cert-manager-operator-controller-manager-55c444bf86-4nknq\" (UID: \"cd6faf27-4405-43fe-a7e8-6ee3b1055c69\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4nknq" Jul 03 10:10:46.186200 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.186161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd6faf27-4405-43fe-a7e8-6ee3b1055c69-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-4nknq\" (UID: \"cd6faf27-4405-43fe-a7e8-6ee3b1055c69\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4nknq" Jul 03 10:10:46.186385 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.186237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqxjm\" (UniqueName: \"kubernetes.io/projected/cd6faf27-4405-43fe-a7e8-6ee3b1055c69-kube-api-access-pqxjm\") pod \"cert-manager-operator-controller-manager-55c444bf86-4nknq\" (UID: \"cd6faf27-4405-43fe-a7e8-6ee3b1055c69\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4nknq" Jul 03 10:10:46.186544 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.186526 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd6faf27-4405-43fe-a7e8-6ee3b1055c69-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-4nknq\" (UID: \"cd6faf27-4405-43fe-a7e8-6ee3b1055c69\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4nknq" Jul 03 10:10:46.197593 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.197562 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqxjm\" (UniqueName: \"kubernetes.io/projected/cd6faf27-4405-43fe-a7e8-6ee3b1055c69-kube-api-access-pqxjm\") pod \"cert-manager-operator-controller-manager-55c444bf86-4nknq\" (UID: \"cd6faf27-4405-43fe-a7e8-6ee3b1055c69\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4nknq" Jul 03 10:10:46.375612 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.375575 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4nknq" Jul 03 10:10:46.415559 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.415524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:10:46.503441 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.503414 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4nknq"] Jul 03 10:10:46.506198 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:10:46.506176 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6faf27_4405_43fe_a7e8_6ee3b1055c69.slice/crio-916eda45f6782e18aca602e4a76e6c9c282bb96aa6c9687cce7a4731009cbc5d WatchSource:0}: Error finding container 916eda45f6782e18aca602e4a76e6c9c282bb96aa6c9687cce7a4731009cbc5d: Status 404 returned error can't find the container with id 916eda45f6782e18aca602e4a76e6c9c282bb96aa6c9687cce7a4731009cbc5d Jul 03 10:10:46.508835 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.508814 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 03 10:10:46.679861 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:46.679776 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4nknq" event={"ID":"cd6faf27-4405-43fe-a7e8-6ee3b1055c69","Type":"ContainerStarted","Data":"916eda45f6782e18aca602e4a76e6c9c282bb96aa6c9687cce7a4731009cbc5d"} Jul 03 10:10:47.723748 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:47.723695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:10:47.870666 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:47.870629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:10:49.694484 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:49.694446 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4nknq" event={"ID":"cd6faf27-4405-43fe-a7e8-6ee3b1055c69","Type":"ContainerStarted","Data":"ab6f331ca8e6b097e0c6bcb39e536308ca9cbc86a4308db4f417c590fa224c3b"} Jul 03 10:10:49.723306 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:49.723242 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-4nknq" podStartSLOduration=1.19432469 podStartE2EDuration="3.723204721s" totalImagesPullingTime="2.528880031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:10:49.720411887 +0000 UTC m=+320.758558496" watchObservedRunningTime="2026-07-03 10:10:49.723204721 +0000 UTC m=+320.761351329" Jul 03 10:10:49.983447 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:49.983337 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:10:50.099243 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:50.099195 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:10:50.279470 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:50.279380 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:10:52.364515 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:52.364480 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:10:52.769015 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:52.768915 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:10:53.098953 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:53.098922 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:10:54.524729 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:54.524685 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:10:54.624283 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:54.624212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:10:59.863041 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:10:59.863006 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:00.110181 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:00.110136 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:00.210992 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:00.210904 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:05.875445 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:05.875398 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:05.975704 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:05.975668 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:07.947099 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:07.947025 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:09.401241 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:09.401192 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:09.916181 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:09.916146 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:20.501707 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:20.501671 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:20.597998 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:20.597964 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:21.116338 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.116259 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:21.403817 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.403732 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:21.505532 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.505500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:21.613889 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.613858 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf"] Jul 03 10:11:21.616810 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.616794 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" Jul 03 10:11:21.627260 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.627240 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 03 10:11:21.627656 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.627640 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hhdl4\"" Jul 03 10:11:21.627893 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.627877 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 03 10:11:21.628544 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.628529 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 03 10:11:21.656376 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.656312 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 03 10:11:21.700281 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.700245 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf"] Jul 03 10:11:21.756333 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.756303 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/07f9e9d4-65d3-4b97-a4eb-a88c47a707f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789547c8f-5trlf\" (UID: \"07f9e9d4-65d3-4b97-a4eb-a88c47a707f8\") " pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" Jul 03 10:11:21.756498 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.756358 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/07f9e9d4-65d3-4b97-a4eb-a88c47a707f8-manifests\") pod \"opendatahub-operator-controller-manager-789547c8f-5trlf\" (UID: \"07f9e9d4-65d3-4b97-a4eb-a88c47a707f8\") " pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" Jul 03 10:11:21.756498 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.756416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07f9e9d4-65d3-4b97-a4eb-a88c47a707f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-789547c8f-5trlf\" (UID: \"07f9e9d4-65d3-4b97-a4eb-a88c47a707f8\") " pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" Jul 03 10:11:21.756498 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.756445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn2qc\" (UniqueName: \"kubernetes.io/projected/07f9e9d4-65d3-4b97-a4eb-a88c47a707f8-kube-api-access-vn2qc\") pod \"opendatahub-operator-controller-manager-789547c8f-5trlf\" (UID: \"07f9e9d4-65d3-4b97-a4eb-a88c47a707f8\") " pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" Jul 03 10:11:21.756498 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.756484 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07f9e9d4-65d3-4b97-a4eb-a88c47a707f8-tmp\") pod \"opendatahub-operator-controller-manager-789547c8f-5trlf\" (UID: \"07f9e9d4-65d3-4b97-a4eb-a88c47a707f8\") " pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" Jul 03 10:11:21.856988 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.856953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/07f9e9d4-65d3-4b97-a4eb-a88c47a707f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789547c8f-5trlf\" (UID: \"07f9e9d4-65d3-4b97-a4eb-a88c47a707f8\") " pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" Jul 03 10:11:21.857156 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.856997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/07f9e9d4-65d3-4b97-a4eb-a88c47a707f8-manifests\") pod \"opendatahub-operator-controller-manager-789547c8f-5trlf\" (UID: \"07f9e9d4-65d3-4b97-a4eb-a88c47a707f8\") " pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" Jul 03 10:11:21.857156 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.857019 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07f9e9d4-65d3-4b97-a4eb-a88c47a707f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-789547c8f-5trlf\" (UID: \"07f9e9d4-65d3-4b97-a4eb-a88c47a707f8\") " pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" Jul 03 10:11:21.857156 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.857117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn2qc\" (UniqueName: \"kubernetes.io/projected/07f9e9d4-65d3-4b97-a4eb-a88c47a707f8-kube-api-access-vn2qc\") pod \"opendatahub-operator-controller-manager-789547c8f-5trlf\" (UID: \"07f9e9d4-65d3-4b97-a4eb-a88c47a707f8\") " pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" Jul 03 10:11:21.857410 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.857167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07f9e9d4-65d3-4b97-a4eb-a88c47a707f8-tmp\") pod \"opendatahub-operator-controller-manager-789547c8f-5trlf\" (UID: \"07f9e9d4-65d3-4b97-a4eb-a88c47a707f8\") " pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" Jul 03 10:11:21.857465 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.857404 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/07f9e9d4-65d3-4b97-a4eb-a88c47a707f8-manifests\") pod \"opendatahub-operator-controller-manager-789547c8f-5trlf\" (UID: \"07f9e9d4-65d3-4b97-a4eb-a88c47a707f8\") " pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" Jul 03 10:11:21.859344 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.859322 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07f9e9d4-65d3-4b97-a4eb-a88c47a707f8-tmp\") pod \"opendatahub-operator-controller-manager-789547c8f-5trlf\" (UID: \"07f9e9d4-65d3-4b97-a4eb-a88c47a707f8\") " pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" Jul 03 10:11:21.859681 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.859660 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/07f9e9d4-65d3-4b97-a4eb-a88c47a707f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-789547c8f-5trlf\" (UID: \"07f9e9d4-65d3-4b97-a4eb-a88c47a707f8\") " pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" Jul 03 10:11:21.859724 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.859669 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07f9e9d4-65d3-4b97-a4eb-a88c47a707f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-789547c8f-5trlf\" (UID: \"07f9e9d4-65d3-4b97-a4eb-a88c47a707f8\") " pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" Jul 03 10:11:21.891199 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.891167 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn2qc\" (UniqueName: \"kubernetes.io/projected/07f9e9d4-65d3-4b97-a4eb-a88c47a707f8-kube-api-access-vn2qc\") pod \"opendatahub-operator-controller-manager-789547c8f-5trlf\" (UID: \"07f9e9d4-65d3-4b97-a4eb-a88c47a707f8\") " pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" Jul 03 10:11:21.926081 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:21.925995 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" Jul 03 10:11:22.097673 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:22.097646 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf"] Jul 03 10:11:22.100162 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:11:22.100135 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f9e9d4_65d3_4b97_a4eb_a88c47a707f8.slice/crio-2b9e26d6bacb953b3b0f783c0bf796953a7ced6b633b5dea819325af9726debf WatchSource:0}: Error finding container 2b9e26d6bacb953b3b0f783c0bf796953a7ced6b633b5dea819325af9726debf: Status 404 returned error can't find the container with id 2b9e26d6bacb953b3b0f783c0bf796953a7ced6b633b5dea819325af9726debf Jul 03 10:11:22.171497 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:22.171451 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:22.279389 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:22.279297 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:22.382836 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:22.382802 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:22.481083 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:22.481045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:22.572661 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:22.572563 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:22.678394 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:22.678359 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:22.807125 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:22.807092 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" event={"ID":"07f9e9d4-65d3-4b97-a4eb-a88c47a707f8","Type":"ContainerStarted","Data":"2b9e26d6bacb953b3b0f783c0bf796953a7ced6b633b5dea819325af9726debf"} Jul 03 10:11:24.114661 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:24.114624 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:24.240700 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:24.240664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:25.825696 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:25.825651 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" event={"ID":"07f9e9d4-65d3-4b97-a4eb-a88c47a707f8","Type":"ContainerStarted","Data":"672d49069f8bed3e3c0d21c1d8a53b0290f68f35b4ab86b8adaf53d8bb1b61d6"} Jul 03 10:11:26.831110 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:26.831072 2579 generic.go:356] "Generic (PLEG): container finished" podID="07f9e9d4-65d3-4b97-a4eb-a88c47a707f8" containerID="672d49069f8bed3e3c0d21c1d8a53b0290f68f35b4ab86b8adaf53d8bb1b61d6" exitCode=0 Jul 03 10:11:26.831520 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:26.831161 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" event={"ID":"07f9e9d4-65d3-4b97-a4eb-a88c47a707f8","Type":"ContainerDied","Data":"672d49069f8bed3e3c0d21c1d8a53b0290f68f35b4ab86b8adaf53d8bb1b61d6"} Jul 03 10:11:27.838874 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:27.838834 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" event={"ID":"07f9e9d4-65d3-4b97-a4eb-a88c47a707f8","Type":"ContainerStarted","Data":"c1ae87351799af8487cca1de47ef257f106e89bfebcad030b212d60ad18ecd3b"} Jul 03 10:11:27.839287 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:27.838923 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" Jul 03 10:11:27.860715 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:27.860661 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" podStartSLOduration=2.507379686 podStartE2EDuration="6.86064497s" totalImagesPullingTime="3.303684595s" totalInitContainerRuntime="1.049580689s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:11:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:11:27.858705517 +0000 UTC m=+358.896852141" watchObservedRunningTime="2026-07-03 10:11:27.86064497 +0000 UTC m=+358.898791577" Jul 03 10:11:28.130286 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:28.130245 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:28.230957 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:28.230922 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:32.944483 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:32.944445 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:33.058228 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:33.058194 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:38.844986 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:38.844954 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-789547c8f-5trlf" Jul 03 10:11:40.866927 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:40.866893 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:41.073320 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:41.073286 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:41.526869 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:41.526834 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:41.544437 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:41.544405 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5b56494769-gmjkj"] Jul 03 10:11:41.553421 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:41.553397 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b56494769-gmjkj" Jul 03 10:11:41.555883 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:41.555853 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 03 10:11:41.556441 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:41.556418 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 03 10:11:41.568633 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:41.568611 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 03 10:11:41.570619 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:41.570599 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 03 10:11:41.585681 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:41.585653 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b56494769-gmjkj"] Jul 03 10:11:41.722531 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:41.722501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvt8r\" (UniqueName: \"kubernetes.io/projected/170f0891-9d61-4a9e-9211-4fcae92ddc58-kube-api-access-gvt8r\") pod \"kube-auth-proxy-5b56494769-gmjkj\" (UID: \"170f0891-9d61-4a9e-9211-4fcae92ddc58\") " pod="openshift-ingress/kube-auth-proxy-5b56494769-gmjkj" Jul 03 10:11:41.722721 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:41.722558 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/170f0891-9d61-4a9e-9211-4fcae92ddc58-tls-certs\") pod \"kube-auth-proxy-5b56494769-gmjkj\" (UID: \"170f0891-9d61-4a9e-9211-4fcae92ddc58\") " pod="openshift-ingress/kube-auth-proxy-5b56494769-gmjkj" Jul 03 10:11:41.722721 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:41.722577 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/170f0891-9d61-4a9e-9211-4fcae92ddc58-tmp\") pod \"kube-auth-proxy-5b56494769-gmjkj\" (UID: \"170f0891-9d61-4a9e-9211-4fcae92ddc58\") " pod="openshift-ingress/kube-auth-proxy-5b56494769-gmjkj" Jul 03 10:11:41.823944 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:41.823843 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/170f0891-9d61-4a9e-9211-4fcae92ddc58-tmp\") pod \"kube-auth-proxy-5b56494769-gmjkj\" (UID: \"170f0891-9d61-4a9e-9211-4fcae92ddc58\") " pod="openshift-ingress/kube-auth-proxy-5b56494769-gmjkj" Jul 03 10:11:41.823944 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:41.823898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvt8r\" (UniqueName: \"kubernetes.io/projected/170f0891-9d61-4a9e-9211-4fcae92ddc58-kube-api-access-gvt8r\") pod \"kube-auth-proxy-5b56494769-gmjkj\" (UID: \"170f0891-9d61-4a9e-9211-4fcae92ddc58\") " pod="openshift-ingress/kube-auth-proxy-5b56494769-gmjkj" Jul 03 10:11:41.824172 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:41.824052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/170f0891-9d61-4a9e-9211-4fcae92ddc58-tls-certs\") pod \"kube-auth-proxy-5b56494769-gmjkj\" (UID: \"170f0891-9d61-4a9e-9211-4fcae92ddc58\") " pod="openshift-ingress/kube-auth-proxy-5b56494769-gmjkj" Jul 03 10:11:41.824260 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:11:41.824175 2579 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 03 10:11:41.824321 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:11:41.824264 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/170f0891-9d61-4a9e-9211-4fcae92ddc58-tls-certs podName:170f0891-9d61-4a9e-9211-4fcae92ddc58 nodeName:}" failed. No retries permitted until 2026-07-03 10:11:42.324241462 +0000 UTC m=+373.362388059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/170f0891-9d61-4a9e-9211-4fcae92ddc58-tls-certs") pod "kube-auth-proxy-5b56494769-gmjkj" (UID: "170f0891-9d61-4a9e-9211-4fcae92ddc58") : secret "kube-auth-proxy-tls" not found Jul 03 10:11:41.826174 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:41.826145 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/170f0891-9d61-4a9e-9211-4fcae92ddc58-tmp\") pod \"kube-auth-proxy-5b56494769-gmjkj\" (UID: \"170f0891-9d61-4a9e-9211-4fcae92ddc58\") " pod="openshift-ingress/kube-auth-proxy-5b56494769-gmjkj" Jul 03 10:11:41.849579 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:41.849545 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvt8r\" (UniqueName: \"kubernetes.io/projected/170f0891-9d61-4a9e-9211-4fcae92ddc58-kube-api-access-gvt8r\") pod \"kube-auth-proxy-5b56494769-gmjkj\" (UID: \"170f0891-9d61-4a9e-9211-4fcae92ddc58\") " pod="openshift-ingress/kube-auth-proxy-5b56494769-gmjkj" Jul 03 10:11:42.022622 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.022592 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r9nqc"] Jul 03 10:11:42.025961 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.025942 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-r9nqc" Jul 03 10:11:42.027900 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.027869 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 03 10:11:42.028033 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.027963 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2m265\"" Jul 03 10:11:42.038633 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.038605 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r9nqc"] Jul 03 10:11:42.045413 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.045388 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:42.126439 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.126402 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3f3093f-95db-461e-99ae-f8d3153e85bb-cert\") pod \"odh-model-controller-7998c67dc4-r9nqc\" (UID: \"c3f3093f-95db-461e-99ae-f8d3153e85bb\") " pod="opendatahub/odh-model-controller-7998c67dc4-r9nqc" Jul 03 10:11:42.126615 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.126449 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x24p\" (UniqueName: \"kubernetes.io/projected/c3f3093f-95db-461e-99ae-f8d3153e85bb-kube-api-access-8x24p\") pod \"odh-model-controller-7998c67dc4-r9nqc\" (UID: \"c3f3093f-95db-461e-99ae-f8d3153e85bb\") " pod="opendatahub/odh-model-controller-7998c67dc4-r9nqc" Jul 03 10:11:42.151870 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.151826 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:42.227497 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.227459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3f3093f-95db-461e-99ae-f8d3153e85bb-cert\") pod \"odh-model-controller-7998c67dc4-r9nqc\" (UID: \"c3f3093f-95db-461e-99ae-f8d3153e85bb\") " pod="opendatahub/odh-model-controller-7998c67dc4-r9nqc" Jul 03 10:11:42.227687 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.227504 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x24p\" (UniqueName: \"kubernetes.io/projected/c3f3093f-95db-461e-99ae-f8d3153e85bb-kube-api-access-8x24p\") pod \"odh-model-controller-7998c67dc4-r9nqc\" (UID: \"c3f3093f-95db-461e-99ae-f8d3153e85bb\") " pod="opendatahub/odh-model-controller-7998c67dc4-r9nqc" Jul 03 10:11:42.227687 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:11:42.227606 2579 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 03 10:11:42.227687 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:11:42.227670 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3f3093f-95db-461e-99ae-f8d3153e85bb-cert podName:c3f3093f-95db-461e-99ae-f8d3153e85bb nodeName:}" failed. No retries permitted until 2026-07-03 10:11:42.727652781 +0000 UTC m=+373.765799365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c3f3093f-95db-461e-99ae-f8d3153e85bb-cert") pod "odh-model-controller-7998c67dc4-r9nqc" (UID: "c3f3093f-95db-461e-99ae-f8d3153e85bb") : secret "odh-model-controller-webhook-cert" not found Jul 03 10:11:42.236139 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.236117 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x24p\" (UniqueName: \"kubernetes.io/projected/c3f3093f-95db-461e-99ae-f8d3153e85bb-kube-api-access-8x24p\") pod \"odh-model-controller-7998c67dc4-r9nqc\" (UID: \"c3f3093f-95db-461e-99ae-f8d3153e85bb\") " pod="opendatahub/odh-model-controller-7998c67dc4-r9nqc" Jul 03 10:11:42.247022 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.246980 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:42.328202 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.328168 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/170f0891-9d61-4a9e-9211-4fcae92ddc58-tls-certs\") pod \"kube-auth-proxy-5b56494769-gmjkj\" (UID: \"170f0891-9d61-4a9e-9211-4fcae92ddc58\") " pod="openshift-ingress/kube-auth-proxy-5b56494769-gmjkj" Jul 03 10:11:42.330656 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.330627 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/170f0891-9d61-4a9e-9211-4fcae92ddc58-tls-certs\") pod \"kube-auth-proxy-5b56494769-gmjkj\" (UID: \"170f0891-9d61-4a9e-9211-4fcae92ddc58\") " pod="openshift-ingress/kube-auth-proxy-5b56494769-gmjkj" Jul 03 10:11:42.347395 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.347366 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:42.465674 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.465587 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b56494769-gmjkj" Jul 03 10:11:42.624848 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.624816 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b56494769-gmjkj"] Jul 03 10:11:42.627943 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:11:42.627915 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170f0891_9d61_4a9e_9211_4fcae92ddc58.slice/crio-e00521745bf55d9cbc42afa4f92b96ee386b3de5ebcc899a2613fd5f2a140184 WatchSource:0}: Error finding container e00521745bf55d9cbc42afa4f92b96ee386b3de5ebcc899a2613fd5f2a140184: Status 404 returned error can't find the container with id e00521745bf55d9cbc42afa4f92b96ee386b3de5ebcc899a2613fd5f2a140184 Jul 03 10:11:42.730907 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.730812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3f3093f-95db-461e-99ae-f8d3153e85bb-cert\") pod \"odh-model-controller-7998c67dc4-r9nqc\" (UID: \"c3f3093f-95db-461e-99ae-f8d3153e85bb\") " pod="opendatahub/odh-model-controller-7998c67dc4-r9nqc" Jul 03 10:11:42.733194 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.733168 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3f3093f-95db-461e-99ae-f8d3153e85bb-cert\") pod \"odh-model-controller-7998c67dc4-r9nqc\" (UID: \"c3f3093f-95db-461e-99ae-f8d3153e85bb\") " pod="opendatahub/odh-model-controller-7998c67dc4-r9nqc" Jul 03 10:11:42.746188 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.746151 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:42.893273 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.893239 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b56494769-gmjkj" event={"ID":"170f0891-9d61-4a9e-9211-4fcae92ddc58","Type":"ContainerStarted","Data":"e00521745bf55d9cbc42afa4f92b96ee386b3de5ebcc899a2613fd5f2a140184"} Jul 03 10:11:42.936334 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:42.936285 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-r9nqc" Jul 03 10:11:43.032999 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.032962 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:43.094416 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.094387 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r9nqc"] Jul 03 10:11:43.098751 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:11:43.098726 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f3093f_95db_461e_99ae_f8d3153e85bb.slice/crio-fd7ed1121ad4529d7de7316dd16cd9800e1ebe04e4fee7e9d076ad331ad037fa WatchSource:0}: Error finding container fd7ed1121ad4529d7de7316dd16cd9800e1ebe04e4fee7e9d076ad331ad037fa: Status 404 returned error can't find the container with id fd7ed1121ad4529d7de7316dd16cd9800e1ebe04e4fee7e9d076ad331ad037fa Jul 03 10:11:43.151923 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.151884 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-w9t6l"] Jul 03 10:11:43.156202 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.156182 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-w9t6l" Jul 03 10:11:43.158372 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.158351 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-4nqcv\"" Jul 03 10:11:43.158495 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.158368 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 03 10:11:43.176662 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.172078 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-w9t6l"] Jul 03 10:11:43.218534 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.218499 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:43.226989 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.226964 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-468j2"] Jul 03 10:11:43.230167 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.230151 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.232201 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.232177 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 03 10:11:43.232328 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.232181 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 03 10:11:43.232403 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.232263 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vpp7g\"" Jul 03 10:11:43.232615 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.232592 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 03 10:11:43.232703 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.232598 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 03 10:11:43.247346 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.247311 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-468j2"] Jul 03 10:11:43.314233 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.314181 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:43.334759 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.334721 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncf46\" (UniqueName: \"kubernetes.io/projected/c16b6d59-1b7e-4e64-a519-87ffecd207bd-kube-api-access-ncf46\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.334943 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.334777 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c16b6d59-1b7e-4e64-a519-87ffecd207bd-cacerts\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.334943 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.334835 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.334943 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.334870 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.334943 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.334916 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.334943 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.334944 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lbwd\" (UniqueName: \"kubernetes.io/projected/2bee4600-2b04-4cbd-b3fe-518f0ee736bf-kube-api-access-6lbwd\") pod \"model-serving-api-54d89bd5fc-w9t6l\" (UID: \"2bee4600-2b04-4cbd-b3fe-518f0ee736bf\") " pod="opendatahub/model-serving-api-54d89bd5fc-w9t6l" Jul 03 10:11:43.335245 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.334978 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-token\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.335245 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.335063 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bee4600-2b04-4cbd-b3fe-518f0ee736bf-tls-certs\") pod \"model-serving-api-54d89bd5fc-w9t6l\" (UID: \"2bee4600-2b04-4cbd-b3fe-518f0ee736bf\") " pod="opendatahub/model-serving-api-54d89bd5fc-w9t6l" Jul 03 10:11:43.335245 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.335122 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16b6d59-1b7e-4e64-a519-87ffecd207bd-local-certs\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.436103 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.436068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bee4600-2b04-4cbd-b3fe-518f0ee736bf-tls-certs\") pod \"model-serving-api-54d89bd5fc-w9t6l\" (UID: \"2bee4600-2b04-4cbd-b3fe-518f0ee736bf\") " pod="opendatahub/model-serving-api-54d89bd5fc-w9t6l" Jul 03 10:11:43.436307 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.436113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16b6d59-1b7e-4e64-a519-87ffecd207bd-local-certs\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.436307 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.436148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncf46\" (UniqueName: \"kubernetes.io/projected/c16b6d59-1b7e-4e64-a519-87ffecd207bd-kube-api-access-ncf46\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.436307 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.436181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c16b6d59-1b7e-4e64-a519-87ffecd207bd-cacerts\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.436307 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.436204 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.436307 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.436241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.436307 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.436276 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.436307 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.436300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lbwd\" (UniqueName: \"kubernetes.io/projected/2bee4600-2b04-4cbd-b3fe-518f0ee736bf-kube-api-access-6lbwd\") pod \"model-serving-api-54d89bd5fc-w9t6l\" (UID: \"2bee4600-2b04-4cbd-b3fe-518f0ee736bf\") " pod="opendatahub/model-serving-api-54d89bd5fc-w9t6l" Jul 03 10:11:43.436681 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.436336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-token\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.437347 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.437313 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.438841 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.438815 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16b6d59-1b7e-4e64-a519-87ffecd207bd-local-certs\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.439357 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.439332 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bee4600-2b04-4cbd-b3fe-518f0ee736bf-tls-certs\") pod \"model-serving-api-54d89bd5fc-w9t6l\" (UID: \"2bee4600-2b04-4cbd-b3fe-518f0ee736bf\") " pod="opendatahub/model-serving-api-54d89bd5fc-w9t6l" Jul 03 10:11:43.439460 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.439385 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c16b6d59-1b7e-4e64-a519-87ffecd207bd-cacerts\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.439460 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.439445 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.439570 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.439535 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.445439 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.445411 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncf46\" (UniqueName: \"kubernetes.io/projected/c16b6d59-1b7e-4e64-a519-87ffecd207bd-kube-api-access-ncf46\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.445663 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.445636 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-token\") pod \"istiod-openshift-gateway-7574459656-468j2\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.446600 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.446575 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lbwd\" (UniqueName: \"kubernetes.io/projected/2bee4600-2b04-4cbd-b3fe-518f0ee736bf-kube-api-access-6lbwd\") pod \"model-serving-api-54d89bd5fc-w9t6l\" (UID: \"2bee4600-2b04-4cbd-b3fe-518f0ee736bf\") " pod="opendatahub/model-serving-api-54d89bd5fc-w9t6l" Jul 03 10:11:43.449658 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.449633 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:43.468669 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.468636 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-w9t6l" Jul 03 10:11:43.541437 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.540988 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:43.631094 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.631002 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-w9t6l"] Jul 03 10:11:43.635148 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:11:43.635107 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bee4600_2b04_4cbd_b3fe_518f0ee736bf.slice/crio-4eac8b9c9670c71fef0f2e524146906f26d90b905089de69de6de6471adfd9ee WatchSource:0}: Error finding container 4eac8b9c9670c71fef0f2e524146906f26d90b905089de69de6de6471adfd9ee: Status 404 returned error can't find the container with id 4eac8b9c9670c71fef0f2e524146906f26d90b905089de69de6de6471adfd9ee Jul 03 10:11:43.714782 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.714747 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-468j2"] Jul 03 10:11:43.718487 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:11:43.718450 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16b6d59_1b7e_4e64_a519_87ffecd207bd.slice/crio-fdf2b3b47d17c8db8930a791e93f14d0b62fbc90a0318ce0620436b661ca1d69 WatchSource:0}: Error finding container fdf2b3b47d17c8db8930a791e93f14d0b62fbc90a0318ce0620436b661ca1d69: Status 404 returned error can't find the container with id fdf2b3b47d17c8db8930a791e93f14d0b62fbc90a0318ce0620436b661ca1d69 Jul 03 10:11:43.899979 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.899934 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-r9nqc" event={"ID":"c3f3093f-95db-461e-99ae-f8d3153e85bb","Type":"ContainerStarted","Data":"fd7ed1121ad4529d7de7316dd16cd9800e1ebe04e4fee7e9d076ad331ad037fa"} Jul 03 10:11:43.901409 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.901385 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-w9t6l" event={"ID":"2bee4600-2b04-4cbd-b3fe-518f0ee736bf","Type":"ContainerStarted","Data":"4eac8b9c9670c71fef0f2e524146906f26d90b905089de69de6de6471adfd9ee"} Jul 03 10:11:43.902916 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:43.902888 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" event={"ID":"c16b6d59-1b7e-4e64-a519-87ffecd207bd","Type":"ContainerStarted","Data":"fdf2b3b47d17c8db8930a791e93f14d0b62fbc90a0318ce0620436b661ca1d69"} Jul 03 10:11:44.449343 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:44.449301 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:48.516968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:48.516930 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 03 10:11:48.517402 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:48.517009 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 03 10:11:48.930499 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:48.930454 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" event={"ID":"c16b6d59-1b7e-4e64-a519-87ffecd207bd","Type":"ContainerStarted","Data":"a81eb8a0f878c6f04fe5ca2c13a0013e0584c06ee030fab5b8d543bc09b3d7bb"} Jul 03 10:11:48.930730 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:48.930701 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:48.932574 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:48.932522 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b56494769-gmjkj" event={"ID":"170f0891-9d61-4a9e-9211-4fcae92ddc58","Type":"ContainerStarted","Data":"6927a62764ad54f8abdf67bd42f85fed95051189688d96227afb064a231a8dfe"} Jul 03 10:11:48.934512 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:48.934465 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-r9nqc" event={"ID":"c3f3093f-95db-461e-99ae-f8d3153e85bb","Type":"ContainerStarted","Data":"9476bfde7beedb7b5a6b0d91ca18384a9a7733cb7e4be3e1d3cd5de807bf3198"} Jul 03 10:11:48.934628 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:48.934606 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-r9nqc" Jul 03 10:11:48.936490 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:48.936459 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-w9t6l" event={"ID":"2bee4600-2b04-4cbd-b3fe-518f0ee736bf","Type":"ContainerStarted","Data":"21f406d2ac407437ea87a13eee708de15d8e3ef0a24fcba39ae2d2902f2e4ad7"} Jul 03 10:11:48.936895 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:48.936875 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-w9t6l" Jul 03 10:11:48.959698 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:48.959628 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" podStartSLOduration=1.1645147200000001 podStartE2EDuration="5.959612018s" totalImagesPullingTime="4.795097298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:11:48.959200243 +0000 UTC m=+379.997346849" watchObservedRunningTime="2026-07-03 10:11:48.959612018 +0000 UTC m=+379.997758622" Jul 03 10:11:48.977018 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:48.976954 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5b56494769-gmjkj" podStartSLOduration=2.096425178 podStartE2EDuration="7.976931701s" totalImagesPullingTime="5.880506523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:11:48.975264022 +0000 UTC m=+380.013410634" watchObservedRunningTime="2026-07-03 10:11:48.976931701 +0000 UTC m=+380.015078309" Jul 03 10:11:49.027652 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:49.027587 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-w9t6l" podStartSLOduration=1.159134433 podStartE2EDuration="6.02756643s" totalImagesPullingTime="4.868431997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:11:48.993291613 +0000 UTC m=+380.031438225" watchObservedRunningTime="2026-07-03 10:11:49.02756643 +0000 UTC m=+380.065713038" Jul 03 10:11:49.029473 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:49.028855 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-r9nqc" podStartSLOduration=2.6230089 podStartE2EDuration="8.028841818s" totalImagesPullingTime="5.405832918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:11:49.027987266 +0000 UTC m=+380.066133900" watchObservedRunningTime="2026-07-03 10:11:49.028841818 +0000 UTC m=+380.066988425" Jul 03 10:11:49.077747 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:49.077716 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:49.162395 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:49.162346 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:49.903175 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:49.903134 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:49.943123 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:49.943095 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:11:50.276402 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:50.276328 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:50.403286 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:50.403248 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:54.717621 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:54.717590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:54.889113 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:54.889082 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:54.891466 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:54.891443 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-s498m"] Jul 03 10:11:54.899866 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:54.899847 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s498m" Jul 03 10:11:54.903164 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:54.903143 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 03 10:11:54.904459 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:54.904440 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-58hfw\"" Jul 03 10:11:54.910003 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:54.909978 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-s498m"] Jul 03 10:11:54.926678 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:54.926646 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hpnsx"] Jul 03 10:11:54.930537 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:54.930518 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpnsx" Jul 03 10:11:54.933016 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:54.932994 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3467416-d3c8-461e-a5a2-a913adb30885-cert\") pod \"kserve-controller-manager-d6f4b65b8-s498m\" (UID: \"e3467416-d3c8-461e-a5a2-a913adb30885\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s498m" Jul 03 10:11:54.933106 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:54.933038 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25j6b\" (UniqueName: \"kubernetes.io/projected/e3467416-d3c8-461e-a5a2-a913adb30885-kube-api-access-25j6b\") pod \"kserve-controller-manager-d6f4b65b8-s498m\" (UID: \"e3467416-d3c8-461e-a5a2-a913adb30885\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s498m" Jul 03 10:11:54.936061 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:54.936042 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 03 10:11:54.940959 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:54.940939 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-fqw5t\"" Jul 03 10:11:54.958176 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:54.958154 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hpnsx"] Jul 03 10:11:55.034596 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:55.034500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25j6b\" (UniqueName: \"kubernetes.io/projected/e3467416-d3c8-461e-a5a2-a913adb30885-kube-api-access-25j6b\") pod \"kserve-controller-manager-d6f4b65b8-s498m\" (UID: \"e3467416-d3c8-461e-a5a2-a913adb30885\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s498m" Jul 03 10:11:55.034596 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:55.034565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc5td\" (UniqueName: \"kubernetes.io/projected/ce204869-bbec-4287-9139-a6c80de9fa81-kube-api-access-jc5td\") pod \"llmisvc-controller-manager-777c6bd7d5-hpnsx\" (UID: \"ce204869-bbec-4287-9139-a6c80de9fa81\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpnsx" Jul 03 10:11:55.034802 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:55.034636 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce204869-bbec-4287-9139-a6c80de9fa81-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hpnsx\" (UID: \"ce204869-bbec-4287-9139-a6c80de9fa81\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpnsx" Jul 03 10:11:55.034802 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:55.034701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3467416-d3c8-461e-a5a2-a913adb30885-cert\") pod \"kserve-controller-manager-d6f4b65b8-s498m\" (UID: \"e3467416-d3c8-461e-a5a2-a913adb30885\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s498m" Jul 03 10:11:55.037308 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:55.037285 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3467416-d3c8-461e-a5a2-a913adb30885-cert\") pod \"kserve-controller-manager-d6f4b65b8-s498m\" (UID: \"e3467416-d3c8-461e-a5a2-a913adb30885\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s498m" Jul 03 10:11:55.051430 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:55.051400 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25j6b\" (UniqueName: \"kubernetes.io/projected/e3467416-d3c8-461e-a5a2-a913adb30885-kube-api-access-25j6b\") pod \"kserve-controller-manager-d6f4b65b8-s498m\" (UID: \"e3467416-d3c8-461e-a5a2-a913adb30885\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-s498m" Jul 03 10:11:55.130486 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:55.130449 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:55.135606 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:55.135578 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce204869-bbec-4287-9139-a6c80de9fa81-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hpnsx\" (UID: \"ce204869-bbec-4287-9139-a6c80de9fa81\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpnsx" Jul 03 10:11:55.135754 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:55.135716 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc5td\" (UniqueName: \"kubernetes.io/projected/ce204869-bbec-4287-9139-a6c80de9fa81-kube-api-access-jc5td\") pod \"llmisvc-controller-manager-777c6bd7d5-hpnsx\" (UID: \"ce204869-bbec-4287-9139-a6c80de9fa81\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpnsx" Jul 03 10:11:55.138261 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:55.138239 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce204869-bbec-4287-9139-a6c80de9fa81-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-hpnsx\" (UID: \"ce204869-bbec-4287-9139-a6c80de9fa81\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpnsx" Jul 03 10:11:55.151888 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:55.151854 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc5td\" (UniqueName: \"kubernetes.io/projected/ce204869-bbec-4287-9139-a6c80de9fa81-kube-api-access-jc5td\") pod \"llmisvc-controller-manager-777c6bd7d5-hpnsx\" (UID: \"ce204869-bbec-4287-9139-a6c80de9fa81\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpnsx" Jul 03 10:11:55.210362 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:55.210328 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s498m" Jul 03 10:11:55.240083 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:55.240045 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpnsx" Jul 03 10:11:55.283342 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:55.281939 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:55.390005 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:55.389976 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-s498m"] Jul 03 10:11:55.395656 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:11:55.395631 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3467416_d3c8_461e_a5a2_a913adb30885.slice/crio-a53eaabd1d71ac7b60f611a72aec5df10ac3ca884b48208e723c51599e1a3628 WatchSource:0}: Error finding container a53eaabd1d71ac7b60f611a72aec5df10ac3ca884b48208e723c51599e1a3628: Status 404 returned error can't find the container with id a53eaabd1d71ac7b60f611a72aec5df10ac3ca884b48208e723c51599e1a3628 Jul 03 10:11:55.416308 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:55.416287 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-hpnsx"] Jul 03 10:11:55.417782 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:11:55.417754 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podce204869_bbec_4287_9139_a6c80de9fa81.slice/crio-0cfd74ee1ab18bd62fe64f0c6eec108617278ec353423ed252ba60e99642a6cd WatchSource:0}: Error finding container 0cfd74ee1ab18bd62fe64f0c6eec108617278ec353423ed252ba60e99642a6cd: Status 404 returned error can't find the container with id 0cfd74ee1ab18bd62fe64f0c6eec108617278ec353423ed252ba60e99642a6cd Jul 03 10:11:55.484451 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:55.484414 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:11:55.967052 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:55.967018 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s498m" event={"ID":"e3467416-d3c8-461e-a5a2-a913adb30885","Type":"ContainerStarted","Data":"a53eaabd1d71ac7b60f611a72aec5df10ac3ca884b48208e723c51599e1a3628"} Jul 03 10:11:55.968342 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:55.968314 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpnsx" event={"ID":"ce204869-bbec-4287-9139-a6c80de9fa81","Type":"ContainerStarted","Data":"0cfd74ee1ab18bd62fe64f0c6eec108617278ec353423ed252ba60e99642a6cd"} Jul 03 10:11:58.983766 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:58.983730 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpnsx" event={"ID":"ce204869-bbec-4287-9139-a6c80de9fa81","Type":"ContainerStarted","Data":"eea4693146bcd9029788395a204f4bd291c9b7a2d2fd54c1ff45cc6424e96e52"} Jul 03 10:11:58.984275 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:58.983810 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpnsx" Jul 03 10:11:58.985265 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:58.985237 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s498m" event={"ID":"e3467416-d3c8-461e-a5a2-a913adb30885","Type":"ContainerStarted","Data":"b4b3643ac4cc048c011bd3cc019a68e84640a7c0c9c4abe73539893a433d34b0"} Jul 03 10:11:58.985393 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:58.985367 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s498m" Jul 03 10:11:59.004164 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:59.004119 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpnsx" podStartSLOduration=1.551556729 podStartE2EDuration="5.004107542s" totalImagesPullingTime="3.452550813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:11:59.003732723 +0000 UTC m=+390.041879329" watchObservedRunningTime="2026-07-03 10:11:59.004107542 +0000 UTC m=+390.042254127" Jul 03 10:11:59.024840 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:59.024792 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s498m" podStartSLOduration=1.554274315 podStartE2EDuration="5.0247774s" totalImagesPullingTime="3.470503085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:11:59.023200683 +0000 UTC m=+390.061347300" watchObservedRunningTime="2026-07-03 10:11:59.0247774 +0000 UTC m=+390.062924007" Jul 03 10:11:59.943855 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:11:59.943825 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-r9nqc" Jul 03 10:12:00.949805 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:00.949775 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-w9t6l" Jul 03 10:12:05.434687 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.434651 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-468j2"] Jul 03 10:12:05.435061 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.434829 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" podUID="c16b6d59-1b7e-4e64-a519-87ffecd207bd" containerName="discovery" containerID="cri-o://a81eb8a0f878c6f04fe5ca2c13a0013e0584c06ee030fab5b8d543bc09b3d7bb" gracePeriod=30 Jul 03 10:12:05.678635 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.678607 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:12:05.725979 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.725897 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-kubeconfig\") pod \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " Jul 03 10:12:05.725979 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.725937 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncf46\" (UniqueName: \"kubernetes.io/projected/c16b6d59-1b7e-4e64-a519-87ffecd207bd-kube-api-access-ncf46\") pod \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " Jul 03 10:12:05.726189 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.725987 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-token\") pod \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " Jul 03 10:12:05.726189 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.726019 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-csr-dns-cert\") pod \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " Jul 03 10:12:05.726189 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.726043 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16b6d59-1b7e-4e64-a519-87ffecd207bd-local-certs\") pod \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " Jul 03 10:12:05.726189 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.726098 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-csr-ca-configmap\") pod \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " Jul 03 10:12:05.726189 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.726126 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c16b6d59-1b7e-4e64-a519-87ffecd207bd-cacerts\") pod \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\" (UID: \"c16b6d59-1b7e-4e64-a519-87ffecd207bd\") " Jul 03 10:12:05.726561 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.726529 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "c16b6d59-1b7e-4e64-a519-87ffecd207bd" (UID: "c16b6d59-1b7e-4e64-a519-87ffecd207bd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 10:12:05.726754 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.726714 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-csr-ca-configmap\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:12:05.728461 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.728419 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "c16b6d59-1b7e-4e64-a519-87ffecd207bd" (UID: "c16b6d59-1b7e-4e64-a519-87ffecd207bd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 10:12:05.728461 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.728445 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-token" (OuterVolumeSpecName: "istio-token") pod "c16b6d59-1b7e-4e64-a519-87ffecd207bd" (UID: "c16b6d59-1b7e-4e64-a519-87ffecd207bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 10:12:05.728707 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.728673 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16b6d59-1b7e-4e64-a519-87ffecd207bd-kube-api-access-ncf46" (OuterVolumeSpecName: "kube-api-access-ncf46") pod "c16b6d59-1b7e-4e64-a519-87ffecd207bd" (UID: "c16b6d59-1b7e-4e64-a519-87ffecd207bd"). InnerVolumeSpecName "kube-api-access-ncf46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 10:12:05.728817 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.728712 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "c16b6d59-1b7e-4e64-a519-87ffecd207bd" (UID: "c16b6d59-1b7e-4e64-a519-87ffecd207bd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 10:12:05.728868 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.728837 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c16b6d59-1b7e-4e64-a519-87ffecd207bd-cacerts" (OuterVolumeSpecName: "cacerts") pod "c16b6d59-1b7e-4e64-a519-87ffecd207bd" (UID: "c16b6d59-1b7e-4e64-a519-87ffecd207bd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 10:12:05.728914 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.728898 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16b6d59-1b7e-4e64-a519-87ffecd207bd-local-certs" (OuterVolumeSpecName: "local-certs") pod "c16b6d59-1b7e-4e64-a519-87ffecd207bd" (UID: "c16b6d59-1b7e-4e64-a519-87ffecd207bd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:12:05.827491 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.827444 2579 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-kubeconfig\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:12:05.827491 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.827483 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncf46\" (UniqueName: \"kubernetes.io/projected/c16b6d59-1b7e-4e64-a519-87ffecd207bd-kube-api-access-ncf46\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:12:05.827491 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.827495 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-token\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:12:05.827491 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.827506 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c16b6d59-1b7e-4e64-a519-87ffecd207bd-istio-csr-dns-cert\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:12:05.827748 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.827514 2579 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c16b6d59-1b7e-4e64-a519-87ffecd207bd-local-certs\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:12:05.827748 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:05.827524 2579 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c16b6d59-1b7e-4e64-a519-87ffecd207bd-cacerts\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:12:06.016488 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:06.016395 2579 generic.go:356] "Generic (PLEG): container finished" podID="c16b6d59-1b7e-4e64-a519-87ffecd207bd" containerID="a81eb8a0f878c6f04fe5ca2c13a0013e0584c06ee030fab5b8d543bc09b3d7bb" exitCode=0 Jul 03 10:12:06.016488 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:06.016446 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" Jul 03 10:12:06.016689 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:06.016479 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" event={"ID":"c16b6d59-1b7e-4e64-a519-87ffecd207bd","Type":"ContainerDied","Data":"a81eb8a0f878c6f04fe5ca2c13a0013e0584c06ee030fab5b8d543bc09b3d7bb"} Jul 03 10:12:06.016689 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:06.016530 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-468j2" event={"ID":"c16b6d59-1b7e-4e64-a519-87ffecd207bd","Type":"ContainerDied","Data":"fdf2b3b47d17c8db8930a791e93f14d0b62fbc90a0318ce0620436b661ca1d69"} Jul 03 10:12:06.016689 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:06.016551 2579 scope.go:117] "RemoveContainer" containerID="a81eb8a0f878c6f04fe5ca2c13a0013e0584c06ee030fab5b8d543bc09b3d7bb" Jul 03 10:12:06.025325 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:06.025305 2579 scope.go:117] "RemoveContainer" containerID="a81eb8a0f878c6f04fe5ca2c13a0013e0584c06ee030fab5b8d543bc09b3d7bb" Jul 03 10:12:06.025577 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:12:06.025552 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a81eb8a0f878c6f04fe5ca2c13a0013e0584c06ee030fab5b8d543bc09b3d7bb\": container with ID starting with a81eb8a0f878c6f04fe5ca2c13a0013e0584c06ee030fab5b8d543bc09b3d7bb not found: ID does not exist" containerID="a81eb8a0f878c6f04fe5ca2c13a0013e0584c06ee030fab5b8d543bc09b3d7bb" Jul 03 10:12:06.025662 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:06.025581 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a81eb8a0f878c6f04fe5ca2c13a0013e0584c06ee030fab5b8d543bc09b3d7bb"} err="failed to get container status \"a81eb8a0f878c6f04fe5ca2c13a0013e0584c06ee030fab5b8d543bc09b3d7bb\": rpc error: code = NotFound desc = could not find container \"a81eb8a0f878c6f04fe5ca2c13a0013e0584c06ee030fab5b8d543bc09b3d7bb\": container with ID starting with a81eb8a0f878c6f04fe5ca2c13a0013e0584c06ee030fab5b8d543bc09b3d7bb not found: ID does not exist" Jul 03 10:12:06.038199 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:06.038168 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-468j2"] Jul 03 10:12:06.042903 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:06.042881 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-468j2"] Jul 03 10:12:07.531274 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:07.531238 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c16b6d59-1b7e-4e64-a519-87ffecd207bd" path="/var/lib/kubelet/pods/c16b6d59-1b7e-4e64-a519-87ffecd207bd/volumes" Jul 03 10:12:29.991627 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:29.991593 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-hpnsx" Jul 03 10:12:29.994562 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:29.994540 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-s498m" Jul 03 10:12:36.614457 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:36.614418 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:12:36.681335 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:36.681303 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:12:42.252949 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.252912 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:12:42.353871 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.353835 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:12:42.361912 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.361881 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-clhlf"] Jul 03 10:12:42.362364 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.362346 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c16b6d59-1b7e-4e64-a519-87ffecd207bd" containerName="discovery" Jul 03 10:12:42.362418 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.362369 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c16b6d59-1b7e-4e64-a519-87ffecd207bd" containerName="discovery" Jul 03 10:12:42.362458 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.362438 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c16b6d59-1b7e-4e64-a519-87ffecd207bd" containerName="discovery" Jul 03 10:12:42.369343 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.369321 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-clhlf" Jul 03 10:12:42.371666 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.371643 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 03 10:12:42.372309 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.372288 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 03 10:12:42.372368 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.372294 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-l26lg\"" Jul 03 10:12:42.375808 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.375715 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-clhlf"] Jul 03 10:12:42.449863 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.449829 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:12:42.525581 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.525498 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8qk8\" (UniqueName: \"kubernetes.io/projected/5ae06c78-b290-4d90-9463-ef89d06cea3e-kube-api-access-g8qk8\") pod \"kuadrant-operator-catalog-clhlf\" (UID: \"5ae06c78-b290-4d90-9463-ef89d06cea3e\") " pod="kuadrant-system/kuadrant-operator-catalog-clhlf" Jul 03 10:12:42.626519 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.626484 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8qk8\" (UniqueName: \"kubernetes.io/projected/5ae06c78-b290-4d90-9463-ef89d06cea3e-kube-api-access-g8qk8\") pod \"kuadrant-operator-catalog-clhlf\" (UID: \"5ae06c78-b290-4d90-9463-ef89d06cea3e\") " pod="kuadrant-system/kuadrant-operator-catalog-clhlf" Jul 03 10:12:42.635847 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.635819 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8qk8\" (UniqueName: \"kubernetes.io/projected/5ae06c78-b290-4d90-9463-ef89d06cea3e-kube-api-access-g8qk8\") pod \"kuadrant-operator-catalog-clhlf\" (UID: \"5ae06c78-b290-4d90-9463-ef89d06cea3e\") " pod="kuadrant-system/kuadrant-operator-catalog-clhlf" Jul 03 10:12:42.679967 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.679934 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-clhlf" Jul 03 10:12:42.811943 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.811917 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-clhlf"] Jul 03 10:12:42.814416 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:12:42.814379 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae06c78_b290_4d90_9463_ef89d06cea3e.slice/crio-1df4cce00da3433aa216df48cb7d66678846d8c59a33fa4670f0e0443214857d WatchSource:0}: Error finding container 1df4cce00da3433aa216df48cb7d66678846d8c59a33fa4670f0e0443214857d: Status 404 returned error can't find the container with id 1df4cce00da3433aa216df48cb7d66678846d8c59a33fa4670f0e0443214857d Jul 03 10:12:42.851256 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.851205 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-clhlf"] Jul 03 10:12:42.920359 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.920328 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kw9d4"] Jul 03 10:12:42.923183 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.923166 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kw9d4" Jul 03 10:12:42.934791 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:42.934765 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kw9d4"] Jul 03 10:12:43.030061 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:43.030030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvtt8\" (UniqueName: \"kubernetes.io/projected/9a06a4c9-5506-4190-b7b2-af804d7cc261-kube-api-access-wvtt8\") pod \"kuadrant-operator-catalog-kw9d4\" (UID: \"9a06a4c9-5506-4190-b7b2-af804d7cc261\") " pod="kuadrant-system/kuadrant-operator-catalog-kw9d4" Jul 03 10:12:43.131301 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:43.131271 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvtt8\" (UniqueName: \"kubernetes.io/projected/9a06a4c9-5506-4190-b7b2-af804d7cc261-kube-api-access-wvtt8\") pod \"kuadrant-operator-catalog-kw9d4\" (UID: \"9a06a4c9-5506-4190-b7b2-af804d7cc261\") " pod="kuadrant-system/kuadrant-operator-catalog-kw9d4" Jul 03 10:12:43.140488 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:43.140454 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvtt8\" (UniqueName: \"kubernetes.io/projected/9a06a4c9-5506-4190-b7b2-af804d7cc261-kube-api-access-wvtt8\") pod \"kuadrant-operator-catalog-kw9d4\" (UID: \"9a06a4c9-5506-4190-b7b2-af804d7cc261\") " pod="kuadrant-system/kuadrant-operator-catalog-kw9d4" Jul 03 10:12:43.154107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:43.154062 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:12:43.166824 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:43.166799 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-clhlf" event={"ID":"5ae06c78-b290-4d90-9463-ef89d06cea3e","Type":"ContainerStarted","Data":"1df4cce00da3433aa216df48cb7d66678846d8c59a33fa4670f0e0443214857d"} Jul 03 10:12:43.232386 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:43.232355 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-kw9d4" Jul 03 10:12:43.356436 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:43.356374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:12:43.380439 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:43.380409 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-kw9d4"] Jul 03 10:12:43.454752 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:12:43.454712 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a06a4c9_5506_4190_b7b2_af804d7cc261.slice/crio-0fe371ff98ba4a15da39ff3844f432b744a405b32c4f5edf5975c1c05c1e5221 WatchSource:0}: Error finding container 0fe371ff98ba4a15da39ff3844f432b744a405b32c4f5edf5975c1c05c1e5221: Status 404 returned error can't find the container with id 0fe371ff98ba4a15da39ff3844f432b744a405b32c4f5edf5975c1c05c1e5221 Jul 03 10:12:44.173415 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:44.173375 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kw9d4" event={"ID":"9a06a4c9-5506-4190-b7b2-af804d7cc261","Type":"ContainerStarted","Data":"0fe371ff98ba4a15da39ff3844f432b744a405b32c4f5edf5975c1c05c1e5221"} Jul 03 10:12:45.179295 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:45.179259 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-clhlf" event={"ID":"5ae06c78-b290-4d90-9463-ef89d06cea3e","Type":"ContainerStarted","Data":"da8110a146321e2860bd8a58d2f3ce8b687f9adbe201da16dc4d4366812987bd"} Jul 03 10:12:45.179295 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:45.179293 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-clhlf" podUID="5ae06c78-b290-4d90-9463-ef89d06cea3e" containerName="registry-server" containerID="cri-o://da8110a146321e2860bd8a58d2f3ce8b687f9adbe201da16dc4d4366812987bd" gracePeriod=2 Jul 03 10:12:45.180690 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:45.180669 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-kw9d4" event={"ID":"9a06a4c9-5506-4190-b7b2-af804d7cc261","Type":"ContainerStarted","Data":"92abbd6becfcb971cef03109bde2041f0cde79b6a297a492f087f1ae630eef51"} Jul 03 10:12:45.194921 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:45.194886 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-clhlf" podStartSLOduration=1.19418609 podStartE2EDuration="3.194870252s" totalImagesPullingTime="2.000684162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:12:45.193760252 +0000 UTC m=+436.231906870" watchObservedRunningTime="2026-07-03 10:12:45.194870252 +0000 UTC m=+436.233016863" Jul 03 10:12:45.211813 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:45.211769 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-kw9d4" podStartSLOduration=1.848110801 podStartE2EDuration="3.211754438s" totalImagesPullingTime="1.363643637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:12:45.210912032 +0000 UTC m=+436.249058638" watchObservedRunningTime="2026-07-03 10:12:45.211754438 +0000 UTC m=+436.249901038" Jul 03 10:12:45.418970 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:45.418946 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-clhlf" Jul 03 10:12:45.551788 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:45.551716 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8qk8\" (UniqueName: \"kubernetes.io/projected/5ae06c78-b290-4d90-9463-ef89d06cea3e-kube-api-access-g8qk8\") pod \"5ae06c78-b290-4d90-9463-ef89d06cea3e\" (UID: \"5ae06c78-b290-4d90-9463-ef89d06cea3e\") " Jul 03 10:12:45.553794 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:45.553771 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae06c78-b290-4d90-9463-ef89d06cea3e-kube-api-access-g8qk8" (OuterVolumeSpecName: "kube-api-access-g8qk8") pod "5ae06c78-b290-4d90-9463-ef89d06cea3e" (UID: "5ae06c78-b290-4d90-9463-ef89d06cea3e"). InnerVolumeSpecName "kube-api-access-g8qk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 10:12:45.652446 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:45.652393 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8qk8\" (UniqueName: \"kubernetes.io/projected/5ae06c78-b290-4d90-9463-ef89d06cea3e-kube-api-access-g8qk8\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:12:46.185275 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:46.185242 2579 generic.go:356] "Generic (PLEG): container finished" podID="5ae06c78-b290-4d90-9463-ef89d06cea3e" containerID="da8110a146321e2860bd8a58d2f3ce8b687f9adbe201da16dc4d4366812987bd" exitCode=0 Jul 03 10:12:46.185731 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:46.185280 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-clhlf" Jul 03 10:12:46.185731 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:46.185330 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-clhlf" event={"ID":"5ae06c78-b290-4d90-9463-ef89d06cea3e","Type":"ContainerDied","Data":"da8110a146321e2860bd8a58d2f3ce8b687f9adbe201da16dc4d4366812987bd"} Jul 03 10:12:46.185731 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:46.185386 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-clhlf" event={"ID":"5ae06c78-b290-4d90-9463-ef89d06cea3e","Type":"ContainerDied","Data":"1df4cce00da3433aa216df48cb7d66678846d8c59a33fa4670f0e0443214857d"} Jul 03 10:12:46.185731 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:46.185419 2579 scope.go:117] "RemoveContainer" containerID="da8110a146321e2860bd8a58d2f3ce8b687f9adbe201da16dc4d4366812987bd" Jul 03 10:12:46.199962 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:46.199943 2579 scope.go:117] "RemoveContainer" containerID="da8110a146321e2860bd8a58d2f3ce8b687f9adbe201da16dc4d4366812987bd" Jul 03 10:12:46.200245 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:12:46.200208 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da8110a146321e2860bd8a58d2f3ce8b687f9adbe201da16dc4d4366812987bd\": container with ID starting with da8110a146321e2860bd8a58d2f3ce8b687f9adbe201da16dc4d4366812987bd not found: ID does not exist" containerID="da8110a146321e2860bd8a58d2f3ce8b687f9adbe201da16dc4d4366812987bd" Jul 03 10:12:46.200292 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:46.200253 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da8110a146321e2860bd8a58d2f3ce8b687f9adbe201da16dc4d4366812987bd"} err="failed to get container status \"da8110a146321e2860bd8a58d2f3ce8b687f9adbe201da16dc4d4366812987bd\": rpc error: code = NotFound desc = could not find container \"da8110a146321e2860bd8a58d2f3ce8b687f9adbe201da16dc4d4366812987bd\": container with ID starting with da8110a146321e2860bd8a58d2f3ce8b687f9adbe201da16dc4d4366812987bd not found: ID does not exist" Jul 03 10:12:46.208378 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:46.208355 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-clhlf"] Jul 03 10:12:46.214231 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:46.214192 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-clhlf"] Jul 03 10:12:47.529967 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:47.529935 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ae06c78-b290-4d90-9463-ef89d06cea3e" path="/var/lib/kubelet/pods/5ae06c78-b290-4d90-9463-ef89d06cea3e/volumes" Jul 03 10:12:53.232846 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:53.232753 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-kw9d4" Jul 03 10:12:53.233340 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:53.232912 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-kw9d4" Jul 03 10:12:53.255491 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:53.255456 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-kw9d4" Jul 03 10:12:54.239175 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:12:54.239143 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-kw9d4" Jul 03 10:13:07.277834 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:07.277800 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:07.340948 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:07.340916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:11.109486 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:11.109437 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:13.746585 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:13.746550 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc"] Jul 03 10:13:13.747028 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:13.746872 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ae06c78-b290-4d90-9463-ef89d06cea3e" containerName="registry-server" Jul 03 10:13:13.747028 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:13.746884 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ae06c78-b290-4d90-9463-ef89d06cea3e" containerName="registry-server" Jul 03 10:13:13.747028 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:13.746945 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ae06c78-b290-4d90-9463-ef89d06cea3e" containerName="registry-server" Jul 03 10:13:13.750162 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:13.750143 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc" Jul 03 10:13:13.752436 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:13.752417 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-v68nc\"" Jul 03 10:13:13.760745 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:13.760721 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc"] Jul 03 10:13:13.877159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:13.877125 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjp84\" (UniqueName: \"kubernetes.io/projected/dad60682-1906-441d-b94e-8759facedd7a-kube-api-access-hjp84\") pod \"limitador-operator-controller-manager-859d7b78bd-rzzmc\" (UID: \"dad60682-1906-441d-b94e-8759facedd7a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc" Jul 03 10:13:13.977718 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:13.977680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjp84\" (UniqueName: \"kubernetes.io/projected/dad60682-1906-441d-b94e-8759facedd7a-kube-api-access-hjp84\") pod \"limitador-operator-controller-manager-859d7b78bd-rzzmc\" (UID: \"dad60682-1906-441d-b94e-8759facedd7a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc" Jul 03 10:13:13.997844 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:13.997767 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjp84\" (UniqueName: \"kubernetes.io/projected/dad60682-1906-441d-b94e-8759facedd7a-kube-api-access-hjp84\") pod \"limitador-operator-controller-manager-859d7b78bd-rzzmc\" (UID: \"dad60682-1906-441d-b94e-8759facedd7a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc" Jul 03 10:13:14.060584 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:14.060548 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc" Jul 03 10:13:14.203954 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:14.203932 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc"] Jul 03 10:13:14.205430 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:13:14.205404 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad60682_1906_441d_b94e_8759facedd7a.slice/crio-6ccf050ff14eaac8a98658dcf2062cfb2bf483eedf783f28a82985886a761a88 WatchSource:0}: Error finding container 6ccf050ff14eaac8a98658dcf2062cfb2bf483eedf783f28a82985886a761a88: Status 404 returned error can't find the container with id 6ccf050ff14eaac8a98658dcf2062cfb2bf483eedf783f28a82985886a761a88 Jul 03 10:13:14.300584 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:14.300502 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc" event={"ID":"dad60682-1906-441d-b94e-8759facedd7a","Type":"ContainerStarted","Data":"6ccf050ff14eaac8a98658dcf2062cfb2bf483eedf783f28a82985886a761a88"} Jul 03 10:13:14.720393 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:14.720357 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:14.829375 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:14.829344 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:16.309850 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:16.309759 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc" event={"ID":"dad60682-1906-441d-b94e-8759facedd7a","Type":"ContainerStarted","Data":"ba5a45b7f7301597df54cef9a3505d010450e6cf3ccc4ad45ee27c6793c8a091"} Jul 03 10:13:16.310282 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:16.309864 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc" Jul 03 10:13:18.957177 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:18.957122 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc" podStartSLOduration=4.241350161 podStartE2EDuration="5.957103602s" totalImagesPullingTime="1.715753441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:13:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:13:16.335012015 +0000 UTC m=+467.373158622" watchObservedRunningTime="2026-07-03 10:13:18.957103602 +0000 UTC m=+469.995250250" Jul 03 10:13:18.958107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:18.958087 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs"] Jul 03 10:13:18.965873 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:18.965856 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:18.969138 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:18.969121 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mds8w\"" Jul 03 10:13:18.981110 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:18.981087 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs"] Jul 03 10:13:19.019704 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:19.019675 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttp9q\" (UniqueName: \"kubernetes.io/projected/474caa4b-cfc3-4f0f-8973-a043928fc531-kube-api-access-ttp9q\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8qjvs\" (UID: \"474caa4b-cfc3-4f0f-8973-a043928fc531\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:19.019875 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:19.019719 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/474caa4b-cfc3-4f0f-8973-a043928fc531-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8qjvs\" (UID: \"474caa4b-cfc3-4f0f-8973-a043928fc531\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:19.120489 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:19.120451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/474caa4b-cfc3-4f0f-8973-a043928fc531-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8qjvs\" (UID: \"474caa4b-cfc3-4f0f-8973-a043928fc531\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:19.120640 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:19.120551 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttp9q\" (UniqueName: \"kubernetes.io/projected/474caa4b-cfc3-4f0f-8973-a043928fc531-kube-api-access-ttp9q\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8qjvs\" (UID: \"474caa4b-cfc3-4f0f-8973-a043928fc531\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:19.120935 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:19.120898 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/474caa4b-cfc3-4f0f-8973-a043928fc531-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8qjvs\" (UID: \"474caa4b-cfc3-4f0f-8973-a043928fc531\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:19.131698 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:19.131669 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttp9q\" (UniqueName: \"kubernetes.io/projected/474caa4b-cfc3-4f0f-8973-a043928fc531-kube-api-access-ttp9q\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8qjvs\" (UID: \"474caa4b-cfc3-4f0f-8973-a043928fc531\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:19.275123 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:19.275041 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:19.306901 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:19.306869 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:19.414569 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:19.414534 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs"] Jul 03 10:13:19.418364 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:13:19.418324 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474caa4b_cfc3_4f0f_8973_a043928fc531.slice/crio-6e3429cc3630843498fcbc10cbb9b4ea2c2998b93fabb6c77cf8f3e62c391007 WatchSource:0}: Error finding container 6e3429cc3630843498fcbc10cbb9b4ea2c2998b93fabb6c77cf8f3e62c391007: Status 404 returned error can't find the container with id 6e3429cc3630843498fcbc10cbb9b4ea2c2998b93fabb6c77cf8f3e62c391007 Jul 03 10:13:19.780757 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:19.780715 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:20.329060 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:20.329024 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" event={"ID":"474caa4b-cfc3-4f0f-8973-a043928fc531","Type":"ContainerStarted","Data":"6e3429cc3630843498fcbc10cbb9b4ea2c2998b93fabb6c77cf8f3e62c391007"} Jul 03 10:13:21.432902 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:21.432863 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:21.620156 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:21.616949 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-mtd68"] Jul 03 10:13:21.622711 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:21.622683 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mtd68" Jul 03 10:13:21.625365 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:21.625344 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-78rcm\"" Jul 03 10:13:21.637091 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:21.637062 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-mtd68"] Jul 03 10:13:21.746353 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:21.746260 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkn9d\" (UniqueName: \"kubernetes.io/projected/dff227d7-468e-4b1e-aebf-0d0ed417464a-kube-api-access-kkn9d\") pod \"authorino-operator-75cc8d5cb7-mtd68\" (UID: \"dff227d7-468e-4b1e-aebf-0d0ed417464a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-mtd68" Jul 03 10:13:21.846831 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:21.846789 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkn9d\" (UniqueName: \"kubernetes.io/projected/dff227d7-468e-4b1e-aebf-0d0ed417464a-kube-api-access-kkn9d\") pod \"authorino-operator-75cc8d5cb7-mtd68\" (UID: \"dff227d7-468e-4b1e-aebf-0d0ed417464a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-mtd68" Jul 03 10:13:21.863020 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:21.862985 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkn9d\" (UniqueName: \"kubernetes.io/projected/dff227d7-468e-4b1e-aebf-0d0ed417464a-kube-api-access-kkn9d\") pod \"authorino-operator-75cc8d5cb7-mtd68\" (UID: \"dff227d7-468e-4b1e-aebf-0d0ed417464a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-mtd68" Jul 03 10:13:21.932441 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:21.932409 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mtd68" Jul 03 10:13:21.987575 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:21.987541 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:22.086982 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:22.086944 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-mtd68"] Jul 03 10:13:22.088565 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:22.088538 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:22.089644 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:13:22.089618 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff227d7_468e_4b1e_aebf_0d0ed417464a.slice/crio-d1c109968758b092f700b587dd14dc360193cbfe5adc73e0e27d3b15e793806c WatchSource:0}: Error finding container d1c109968758b092f700b587dd14dc360193cbfe5adc73e0e27d3b15e793806c: Status 404 returned error can't find the container with id d1c109968758b092f700b587dd14dc360193cbfe5adc73e0e27d3b15e793806c Jul 03 10:13:22.339842 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:22.339762 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mtd68" event={"ID":"dff227d7-468e-4b1e-aebf-0d0ed417464a","Type":"ContainerStarted","Data":"d1c109968758b092f700b587dd14dc360193cbfe5adc73e0e27d3b15e793806c"} Jul 03 10:13:22.722827 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:22.722790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:23.797952 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:23.797916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:25.583684 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:25.583648 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:26.359546 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:26.359507 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mtd68" event={"ID":"dff227d7-468e-4b1e-aebf-0d0ed417464a","Type":"ContainerStarted","Data":"4b69323d3c74cac56988964de8dbaf4914b72ddcd515129806c7d6b2506431de"} Jul 03 10:13:26.359749 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:26.359680 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mtd68" Jul 03 10:13:26.361059 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:26.361033 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" event={"ID":"474caa4b-cfc3-4f0f-8973-a043928fc531","Type":"ContainerStarted","Data":"6ede4af058599235677bd986ad38371480d5a9dfe81ab2983ea817af9e49befa"} Jul 03 10:13:26.361177 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:26.361161 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:26.444817 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:26.444765 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" podStartSLOduration=2.154174201 podStartE2EDuration="8.444746131s" totalImagesPullingTime="6.29057193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:13:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:13:26.443408225 +0000 UTC m=+477.481554832" watchObservedRunningTime="2026-07-03 10:13:26.444746131 +0000 UTC m=+477.482892740" Jul 03 10:13:26.444971 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:26.444859 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mtd68" podStartSLOduration=1.835526432 podStartE2EDuration="5.444854292s" totalImagesPullingTime="3.60932786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:13:26.409509491 +0000 UTC m=+477.447656098" watchObservedRunningTime="2026-07-03 10:13:26.444854292 +0000 UTC m=+477.483000899" Jul 03 10:13:26.951414 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:26.951382 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:27.072351 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:27.072312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:27.317368 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:27.317288 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc" Jul 03 10:13:28.882316 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:28.882274 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lvdb6"] Jul 03 10:13:28.885684 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:28.885668 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lvdb6" Jul 03 10:13:28.890833 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:28.890808 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 03 10:13:28.891249 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:28.891234 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 03 10:13:28.896120 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:28.896104 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-snd9l\"" Jul 03 10:13:28.917596 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:28.917566 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lvdb6"] Jul 03 10:13:28.922806 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:28.922767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:29.006470 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:29.006434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtkmb\" (UniqueName: \"kubernetes.io/projected/9404fda8-21f8-4527-aadf-0bf0bcce7316-kube-api-access-jtkmb\") pod \"kuadrant-console-plugin-66b5774985-lvdb6\" (UID: \"9404fda8-21f8-4527-aadf-0bf0bcce7316\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lvdb6" Jul 03 10:13:29.006646 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:29.006495 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9404fda8-21f8-4527-aadf-0bf0bcce7316-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lvdb6\" (UID: \"9404fda8-21f8-4527-aadf-0bf0bcce7316\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lvdb6" Jul 03 10:13:29.006646 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:29.006593 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9404fda8-21f8-4527-aadf-0bf0bcce7316-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lvdb6\" (UID: \"9404fda8-21f8-4527-aadf-0bf0bcce7316\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lvdb6" Jul 03 10:13:29.107533 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:29.107491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9404fda8-21f8-4527-aadf-0bf0bcce7316-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lvdb6\" (UID: \"9404fda8-21f8-4527-aadf-0bf0bcce7316\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lvdb6" Jul 03 10:13:29.107697 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:29.107655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9404fda8-21f8-4527-aadf-0bf0bcce7316-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lvdb6\" (UID: \"9404fda8-21f8-4527-aadf-0bf0bcce7316\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lvdb6" Jul 03 10:13:29.107757 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:29.107704 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtkmb\" (UniqueName: \"kubernetes.io/projected/9404fda8-21f8-4527-aadf-0bf0bcce7316-kube-api-access-jtkmb\") pod \"kuadrant-console-plugin-66b5774985-lvdb6\" (UID: \"9404fda8-21f8-4527-aadf-0bf0bcce7316\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lvdb6" Jul 03 10:13:29.107815 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:13:29.107799 2579 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 03 10:13:29.107885 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:13:29.107873 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9404fda8-21f8-4527-aadf-0bf0bcce7316-plugin-serving-cert podName:9404fda8-21f8-4527-aadf-0bf0bcce7316 nodeName:}" failed. No retries permitted until 2026-07-03 10:13:29.607852306 +0000 UTC m=+480.645998906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/9404fda8-21f8-4527-aadf-0bf0bcce7316-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-lvdb6" (UID: "9404fda8-21f8-4527-aadf-0bf0bcce7316") : secret "plugin-serving-cert" not found Jul 03 10:13:29.108114 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:29.108096 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9404fda8-21f8-4527-aadf-0bf0bcce7316-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lvdb6\" (UID: \"9404fda8-21f8-4527-aadf-0bf0bcce7316\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lvdb6" Jul 03 10:13:29.119527 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:29.119497 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtkmb\" (UniqueName: \"kubernetes.io/projected/9404fda8-21f8-4527-aadf-0bf0bcce7316-kube-api-access-jtkmb\") pod \"kuadrant-console-plugin-66b5774985-lvdb6\" (UID: \"9404fda8-21f8-4527-aadf-0bf0bcce7316\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lvdb6" Jul 03 10:13:29.264607 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:29.264515 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:29.611691 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:29.611665 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9404fda8-21f8-4527-aadf-0bf0bcce7316-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lvdb6\" (UID: \"9404fda8-21f8-4527-aadf-0bf0bcce7316\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lvdb6" Jul 03 10:13:29.614115 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:29.614079 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9404fda8-21f8-4527-aadf-0bf0bcce7316-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lvdb6\" (UID: \"9404fda8-21f8-4527-aadf-0bf0bcce7316\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lvdb6" Jul 03 10:13:29.798612 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:29.798582 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-snd9l\"" Jul 03 10:13:29.806691 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:29.806670 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lvdb6" Jul 03 10:13:29.934193 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:29.934165 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lvdb6"] Jul 03 10:13:29.936558 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:13:29.936527 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9404fda8_21f8_4527_aadf_0bf0bcce7316.slice/crio-4d77f036deb35044ef28f6c8bee5be721e0ac09bae6fc695a71749044dfa1b8a WatchSource:0}: Error finding container 4d77f036deb35044ef28f6c8bee5be721e0ac09bae6fc695a71749044dfa1b8a: Status 404 returned error can't find the container with id 4d77f036deb35044ef28f6c8bee5be721e0ac09bae6fc695a71749044dfa1b8a Jul 03 10:13:30.324192 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:30.324108 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:30.382722 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:30.382690 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lvdb6" event={"ID":"9404fda8-21f8-4527-aadf-0bf0bcce7316","Type":"ContainerStarted","Data":"4d77f036deb35044ef28f6c8bee5be721e0ac09bae6fc695a71749044dfa1b8a"} Jul 03 10:13:30.423618 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:30.423589 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:31.055002 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:31.054970 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:32.251176 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:32.251143 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:32.431208 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:32.431158 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:33.153956 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:33.153909 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:33.299932 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:33.299873 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:34.014523 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:34.014494 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:37.368186 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:37.368067 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:37.368688 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:37.368557 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-mtd68" Jul 03 10:13:39.581013 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:39.580978 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:39.869079 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:39.869031 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs"] Jul 03 10:13:39.869317 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:39.869294 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" podUID="474caa4b-cfc3-4f0f-8973-a043928fc531" containerName="manager" containerID="cri-o://6ede4af058599235677bd986ad38371480d5a9dfe81ab2983ea817af9e49befa" gracePeriod=2 Jul 03 10:13:39.879279 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:39.879249 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs"] Jul 03 10:13:39.915479 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:13:39.915425 2579 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"474caa4b-cfc3-4f0f-8973-a043928fc531\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-8qjvs\": pods \"kuadrant-operator-controller-manager-8684c9ff49-8qjvs\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:39.917351 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:39.917326 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc"] Jul 03 10:13:39.917560 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:39.917535 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc" podUID="dad60682-1906-441d-b94e-8759facedd7a" containerName="manager" containerID="cri-o://ba5a45b7f7301597df54cef9a3505d010450e6cf3ccc4ad45ee27c6793c8a091" gracePeriod=2 Jul 03 10:13:39.927378 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:13:39.927345 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8qjvs\" is forbidden: User \"system:node:ip-10-0-143-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-105.ec2.internal' and this object" podUID="474caa4b-cfc3-4f0f-8973-a043928fc531" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:39.932484 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:39.932456 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc"] Jul 03 10:13:39.960841 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:39.960799 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:39.963012 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:39.962975 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq"] Jul 03 10:13:39.963434 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:39.963409 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="474caa4b-cfc3-4f0f-8973-a043928fc531" containerName="manager" Jul 03 10:13:39.963434 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:39.963428 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="474caa4b-cfc3-4f0f-8973-a043928fc531" containerName="manager" Jul 03 10:13:39.963574 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:39.963464 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dad60682-1906-441d-b94e-8759facedd7a" containerName="manager" Jul 03 10:13:39.963574 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:39.963473 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dad60682-1906-441d-b94e-8759facedd7a" containerName="manager" Jul 03 10:13:39.963574 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:39.963541 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dad60682-1906-441d-b94e-8759facedd7a" containerName="manager" Jul 03 10:13:39.963574 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:39.963554 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="474caa4b-cfc3-4f0f-8973-a043928fc531" containerName="manager" Jul 03 10:13:39.968005 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:39.967983 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq" Jul 03 10:13:39.972533 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:13:39.972498 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-vnbnq\" is forbidden: User \"system:node:ip-10-0-143-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-105.ec2.internal' and this object" podUID="6600a0b6-ae91-4f07-9871-7dc2447726e3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq" Jul 03 10:13:39.974816 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:13:39.974656 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8qjvs\" is forbidden: User \"system:node:ip-10-0-143-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-105.ec2.internal' and this object" podUID="474caa4b-cfc3-4f0f-8973-a043928fc531" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:39.976415 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:13:39.976381 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rzzmc\" is forbidden: User \"system:node:ip-10-0-143-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-105.ec2.internal' and this object" podUID="dad60682-1906-441d-b94e-8759facedd7a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc" Jul 03 10:13:40.019702 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:40.019665 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vxnfn"] Jul 03 10:13:40.026443 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:40.026407 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vxnfn" Jul 03 10:13:40.028660 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:13:40.028623 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rzzmc\" is forbidden: User \"system:node:ip-10-0-143-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-105.ec2.internal' and this object" podUID="dad60682-1906-441d-b94e-8759facedd7a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc" Jul 03 10:13:40.055077 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:40.055020 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq"] Jul 03 10:13:40.081985 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:40.081917 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vxnfn"] Jul 03 10:13:40.108033 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:40.107998 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6600a0b6-ae91-4f07-9871-7dc2447726e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vnbnq\" (UID: \"6600a0b6-ae91-4f07-9871-7dc2447726e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq" Jul 03 10:13:40.108249 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:40.108041 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxqxz\" (UniqueName: \"kubernetes.io/projected/6600a0b6-ae91-4f07-9871-7dc2447726e3-kube-api-access-wxqxz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vnbnq\" (UID: \"6600a0b6-ae91-4f07-9871-7dc2447726e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq" Jul 03 10:13:40.145439 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:13:40.145351 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8qjvs\" is forbidden: User \"system:node:ip-10-0-143-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-105.ec2.internal' and this object" podUID="474caa4b-cfc3-4f0f-8973-a043928fc531" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:40.208986 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:40.208937 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6600a0b6-ae91-4f07-9871-7dc2447726e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vnbnq\" (UID: \"6600a0b6-ae91-4f07-9871-7dc2447726e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq" Jul 03 10:13:40.208986 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:40.208992 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cgnp\" (UniqueName: \"kubernetes.io/projected/7e88a64e-c97f-43eb-83c4-583b034f38a9-kube-api-access-6cgnp\") pod \"limitador-operator-controller-manager-859d7b78bd-vxnfn\" (UID: \"7e88a64e-c97f-43eb-83c4-583b034f38a9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vxnfn" Jul 03 10:13:40.209256 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:40.209025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxqxz\" (UniqueName: \"kubernetes.io/projected/6600a0b6-ae91-4f07-9871-7dc2447726e3-kube-api-access-wxqxz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vnbnq\" (UID: \"6600a0b6-ae91-4f07-9871-7dc2447726e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq" Jul 03 10:13:40.209424 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:40.209400 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6600a0b6-ae91-4f07-9871-7dc2447726e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vnbnq\" (UID: \"6600a0b6-ae91-4f07-9871-7dc2447726e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq" Jul 03 10:13:40.213856 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:40.213815 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:40.231018 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:40.230986 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxqxz\" (UniqueName: \"kubernetes.io/projected/6600a0b6-ae91-4f07-9871-7dc2447726e3-kube-api-access-wxqxz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vnbnq\" (UID: \"6600a0b6-ae91-4f07-9871-7dc2447726e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq" Jul 03 10:13:40.295492 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:40.295446 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq" Jul 03 10:13:40.309651 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:40.309616 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cgnp\" (UniqueName: \"kubernetes.io/projected/7e88a64e-c97f-43eb-83c4-583b034f38a9-kube-api-access-6cgnp\") pod \"limitador-operator-controller-manager-859d7b78bd-vxnfn\" (UID: \"7e88a64e-c97f-43eb-83c4-583b034f38a9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vxnfn" Jul 03 10:13:40.324017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:40.323985 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cgnp\" (UniqueName: \"kubernetes.io/projected/7e88a64e-c97f-43eb-83c4-583b034f38a9-kube-api-access-6cgnp\") pod \"limitador-operator-controller-manager-859d7b78bd-vxnfn\" (UID: \"7e88a64e-c97f-43eb-83c4-583b034f38a9\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vxnfn" Jul 03 10:13:40.341298 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:40.341264 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vxnfn" Jul 03 10:13:41.537337 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:41.537299 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:41.687158 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:41.687130 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:43.034319 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:43.034272 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t"] Jul 03 10:13:43.105098 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:43.105042 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t"] Jul 03 10:13:43.105290 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:43.105169 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t" Jul 03 10:13:43.235526 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:43.235490 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nrph\" (UniqueName: \"kubernetes.io/projected/3bb2a19f-d59f-41e0-96e6-f5a90b6e236f-kube-api-access-5nrph\") pod \"kuadrant-operator-controller-manager-86878c769f-nkw2t\" (UID: \"3bb2a19f-d59f-41e0-96e6-f5a90b6e236f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t" Jul 03 10:13:43.235712 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:43.235553 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3bb2a19f-d59f-41e0-96e6-f5a90b6e236f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-nkw2t\" (UID: \"3bb2a19f-d59f-41e0-96e6-f5a90b6e236f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t" Jul 03 10:13:43.336594 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:43.336500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nrph\" (UniqueName: \"kubernetes.io/projected/3bb2a19f-d59f-41e0-96e6-f5a90b6e236f-kube-api-access-5nrph\") pod \"kuadrant-operator-controller-manager-86878c769f-nkw2t\" (UID: \"3bb2a19f-d59f-41e0-96e6-f5a90b6e236f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t" Jul 03 10:13:43.336763 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:43.336602 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3bb2a19f-d59f-41e0-96e6-f5a90b6e236f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-nkw2t\" (UID: \"3bb2a19f-d59f-41e0-96e6-f5a90b6e236f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t" Jul 03 10:13:43.337018 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:43.337000 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3bb2a19f-d59f-41e0-96e6-f5a90b6e236f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-nkw2t\" (UID: \"3bb2a19f-d59f-41e0-96e6-f5a90b6e236f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t" Jul 03 10:13:43.354514 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:43.354483 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nrph\" (UniqueName: \"kubernetes.io/projected/3bb2a19f-d59f-41e0-96e6-f5a90b6e236f-kube-api-access-5nrph\") pod \"kuadrant-operator-controller-manager-86878c769f-nkw2t\" (UID: \"3bb2a19f-d59f-41e0-96e6-f5a90b6e236f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t" Jul 03 10:13:43.416799 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:43.416766 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t" Jul 03 10:13:46.073657 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:46.073611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:46.269611 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:46.269575 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:48.685971 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:48.685942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:48.790678 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:48.790627 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:52.913194 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:52.913147 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq"] Jul 03 10:13:52.998029 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:13:52.998001 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6600a0b6_ae91_4f07_9871_7dc2447726e3.slice/crio-d84ca987fd4a3a58edc755d1e3f1a24a0718477f386a18e4b46fd81a2ca66fd2 WatchSource:0}: Error finding container d84ca987fd4a3a58edc755d1e3f1a24a0718477f386a18e4b46fd81a2ca66fd2: Status 404 returned error can't find the container with id d84ca987fd4a3a58edc755d1e3f1a24a0718477f386a18e4b46fd81a2ca66fd2 Jul 03 10:13:53.025662 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.025631 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:53.027950 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:13:53.027921 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8qjvs\" is forbidden: User \"system:node:ip-10-0-143-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-105.ec2.internal' and this object" podUID="474caa4b-cfc3-4f0f-8973-a043928fc531" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:53.041942 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.041921 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc" Jul 03 10:13:53.044131 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:13:53.044103 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rzzmc\" is forbidden: User \"system:node:ip-10-0-143-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-105.ec2.internal' and this object" podUID="dad60682-1906-441d-b94e-8759facedd7a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc" Jul 03 10:13:53.045711 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:13:53.045686 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8qjvs\" is forbidden: User \"system:node:ip-10-0-143-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-105.ec2.internal' and this object" podUID="474caa4b-cfc3-4f0f-8973-a043928fc531" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:53.126140 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.126120 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjp84\" (UniqueName: \"kubernetes.io/projected/dad60682-1906-441d-b94e-8759facedd7a-kube-api-access-hjp84\") pod \"dad60682-1906-441d-b94e-8759facedd7a\" (UID: \"dad60682-1906-441d-b94e-8759facedd7a\") " Jul 03 10:13:53.126263 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.126169 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttp9q\" (UniqueName: \"kubernetes.io/projected/474caa4b-cfc3-4f0f-8973-a043928fc531-kube-api-access-ttp9q\") pod \"474caa4b-cfc3-4f0f-8973-a043928fc531\" (UID: \"474caa4b-cfc3-4f0f-8973-a043928fc531\") " Jul 03 10:13:53.126263 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.126211 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/474caa4b-cfc3-4f0f-8973-a043928fc531-extensions-socket-volume\") pod \"474caa4b-cfc3-4f0f-8973-a043928fc531\" (UID: \"474caa4b-cfc3-4f0f-8973-a043928fc531\") " Jul 03 10:13:53.126603 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.126579 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474caa4b-cfc3-4f0f-8973-a043928fc531-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "474caa4b-cfc3-4f0f-8973-a043928fc531" (UID: "474caa4b-cfc3-4f0f-8973-a043928fc531"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:13:53.128003 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.127977 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474caa4b-cfc3-4f0f-8973-a043928fc531-kube-api-access-ttp9q" (OuterVolumeSpecName: "kube-api-access-ttp9q") pod "474caa4b-cfc3-4f0f-8973-a043928fc531" (UID: "474caa4b-cfc3-4f0f-8973-a043928fc531"). InnerVolumeSpecName "kube-api-access-ttp9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 10:13:53.128083 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.128048 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad60682-1906-441d-b94e-8759facedd7a-kube-api-access-hjp84" (OuterVolumeSpecName: "kube-api-access-hjp84") pod "dad60682-1906-441d-b94e-8759facedd7a" (UID: "dad60682-1906-441d-b94e-8759facedd7a"). InnerVolumeSpecName "kube-api-access-hjp84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 10:13:53.136158 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.136131 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vxnfn"] Jul 03 10:13:53.138265 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.138239 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t"] Jul 03 10:13:53.144124 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:13:53.144098 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e88a64e_c97f_43eb_83c4_583b034f38a9.slice/crio-501dde9429974b6fdadf078d9d5525d607e1b8c846c24609480776f3099a6fd9 WatchSource:0}: Error finding container 501dde9429974b6fdadf078d9d5525d607e1b8c846c24609480776f3099a6fd9: Status 404 returned error can't find the container with id 501dde9429974b6fdadf078d9d5525d607e1b8c846c24609480776f3099a6fd9 Jul 03 10:13:53.146244 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:13:53.146200 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb2a19f_d59f_41e0_96e6_f5a90b6e236f.slice/crio-456f104da8a9d2553ddb31ede0cfe297ecab0925af6e2330c76fc4f0af4b45ca WatchSource:0}: Error finding container 456f104da8a9d2553ddb31ede0cfe297ecab0925af6e2330c76fc4f0af4b45ca: Status 404 returned error can't find the container with id 456f104da8a9d2553ddb31ede0cfe297ecab0925af6e2330c76fc4f0af4b45ca Jul 03 10:13:53.227454 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.227430 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttp9q\" (UniqueName: \"kubernetes.io/projected/474caa4b-cfc3-4f0f-8973-a043928fc531-kube-api-access-ttp9q\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:13:53.227454 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.227452 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/474caa4b-cfc3-4f0f-8973-a043928fc531-extensions-socket-volume\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:13:53.227581 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.227463 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjp84\" (UniqueName: \"kubernetes.io/projected/dad60682-1906-441d-b94e-8759facedd7a-kube-api-access-hjp84\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:13:53.510709 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.510674 2579 generic.go:356] "Generic (PLEG): container finished" podID="474caa4b-cfc3-4f0f-8973-a043928fc531" containerID="6ede4af058599235677bd986ad38371480d5a9dfe81ab2983ea817af9e49befa" exitCode=0 Jul 03 10:13:53.510899 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.510723 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:53.510899 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.510767 2579 scope.go:117] "RemoveContainer" containerID="6ede4af058599235677bd986ad38371480d5a9dfe81ab2983ea817af9e49befa" Jul 03 10:13:53.512884 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.512844 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vxnfn" event={"ID":"7e88a64e-c97f-43eb-83c4-583b034f38a9","Type":"ContainerStarted","Data":"fc19c67f08ab4e008eed621fd4469abdeb8105958f50741126332521bc0cbe50"} Jul 03 10:13:53.512989 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.512906 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vxnfn" event={"ID":"7e88a64e-c97f-43eb-83c4-583b034f38a9","Type":"ContainerStarted","Data":"501dde9429974b6fdadf078d9d5525d607e1b8c846c24609480776f3099a6fd9"} Jul 03 10:13:53.513062 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:13:53.512985 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8qjvs\" is forbidden: User \"system:node:ip-10-0-143-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-105.ec2.internal' and this object" podUID="474caa4b-cfc3-4f0f-8973-a043928fc531" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:53.513136 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.513068 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vxnfn" Jul 03 10:13:53.514802 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.514778 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lvdb6" event={"ID":"9404fda8-21f8-4527-aadf-0bf0bcce7316","Type":"ContainerStarted","Data":"cc042c531df9367970c33d16ad6d3ff7eff4cedd5ac2da2345578cfbc4a72004"} Jul 03 10:13:53.514923 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:13:53.514876 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rzzmc\" is forbidden: User \"system:node:ip-10-0-143-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-105.ec2.internal' and this object" podUID="dad60682-1906-441d-b94e-8759facedd7a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc" Jul 03 10:13:53.516696 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:13:53.516663 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-rzzmc\" is forbidden: User \"system:node:ip-10-0-143-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-105.ec2.internal' and this object" podUID="dad60682-1906-441d-b94e-8759facedd7a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc" Jul 03 10:13:53.517194 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.517170 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t" event={"ID":"3bb2a19f-d59f-41e0-96e6-f5a90b6e236f","Type":"ContainerStarted","Data":"e203f8fdb51ac94635983e8f1dcbbaf7cb692fc8fe8fba4aa30ee7524bc81393"} Jul 03 10:13:53.517330 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.517244 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t" Jul 03 10:13:53.517330 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.517272 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t" event={"ID":"3bb2a19f-d59f-41e0-96e6-f5a90b6e236f","Type":"ContainerStarted","Data":"456f104da8a9d2553ddb31ede0cfe297ecab0925af6e2330c76fc4f0af4b45ca"} Jul 03 10:13:53.518595 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.518570 2579 generic.go:356] "Generic (PLEG): container finished" podID="dad60682-1906-441d-b94e-8759facedd7a" containerID="ba5a45b7f7301597df54cef9a3505d010450e6cf3ccc4ad45ee27c6793c8a091" exitCode=0 Jul 03 10:13:53.518700 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.518644 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rzzmc" Jul 03 10:13:53.520534 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.520384 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq" event={"ID":"6600a0b6-ae91-4f07-9871-7dc2447726e3","Type":"ContainerStarted","Data":"1e5921bae6ceb50e25c1f5c470bf319192834a01c110b987a28520481342ba6e"} Jul 03 10:13:53.520534 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.520425 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq" event={"ID":"6600a0b6-ae91-4f07-9871-7dc2447726e3","Type":"ContainerStarted","Data":"d84ca987fd4a3a58edc755d1e3f1a24a0718477f386a18e4b46fd81a2ca66fd2"} Jul 03 10:13:53.520850 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.520829 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq" Jul 03 10:13:53.522416 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.522400 2579 scope.go:117] "RemoveContainer" containerID="6ede4af058599235677bd986ad38371480d5a9dfe81ab2983ea817af9e49befa" Jul 03 10:13:53.522647 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:13:53.522629 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ede4af058599235677bd986ad38371480d5a9dfe81ab2983ea817af9e49befa\": container with ID starting with 6ede4af058599235677bd986ad38371480d5a9dfe81ab2983ea817af9e49befa not found: ID does not exist" containerID="6ede4af058599235677bd986ad38371480d5a9dfe81ab2983ea817af9e49befa" Jul 03 10:13:53.522696 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.522653 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ede4af058599235677bd986ad38371480d5a9dfe81ab2983ea817af9e49befa"} err="failed to get container status \"6ede4af058599235677bd986ad38371480d5a9dfe81ab2983ea817af9e49befa\": rpc error: code = NotFound desc = could not find container \"6ede4af058599235677bd986ad38371480d5a9dfe81ab2983ea817af9e49befa\": container with ID starting with 6ede4af058599235677bd986ad38371480d5a9dfe81ab2983ea817af9e49befa not found: ID does not exist" Jul 03 10:13:53.522696 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.522667 2579 scope.go:117] "RemoveContainer" containerID="ba5a45b7f7301597df54cef9a3505d010450e6cf3ccc4ad45ee27c6793c8a091" Jul 03 10:13:53.533985 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.532255 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="474caa4b-cfc3-4f0f-8973-a043928fc531" path="/var/lib/kubelet/pods/474caa4b-cfc3-4f0f-8973-a043928fc531/volumes" Jul 03 10:13:53.533985 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.532934 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dad60682-1906-441d-b94e-8759facedd7a" path="/var/lib/kubelet/pods/dad60682-1906-441d-b94e-8759facedd7a/volumes" Jul 03 10:13:53.533985 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.533554 2579 scope.go:117] "RemoveContainer" containerID="ba5a45b7f7301597df54cef9a3505d010450e6cf3ccc4ad45ee27c6793c8a091" Jul 03 10:13:53.533985 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:13:53.533881 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba5a45b7f7301597df54cef9a3505d010450e6cf3ccc4ad45ee27c6793c8a091\": container with ID starting with ba5a45b7f7301597df54cef9a3505d010450e6cf3ccc4ad45ee27c6793c8a091 not found: ID does not exist" containerID="ba5a45b7f7301597df54cef9a3505d010450e6cf3ccc4ad45ee27c6793c8a091" Jul 03 10:13:53.533985 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.533913 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5a45b7f7301597df54cef9a3505d010450e6cf3ccc4ad45ee27c6793c8a091"} err="failed to get container status \"ba5a45b7f7301597df54cef9a3505d010450e6cf3ccc4ad45ee27c6793c8a091\": rpc error: code = NotFound desc = could not find container \"ba5a45b7f7301597df54cef9a3505d010450e6cf3ccc4ad45ee27c6793c8a091\": container with ID starting with ba5a45b7f7301597df54cef9a3505d010450e6cf3ccc4ad45ee27c6793c8a091 not found: ID does not exist" Jul 03 10:13:53.539457 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.539414 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vxnfn" podStartSLOduration=14.539401193 podStartE2EDuration="14.539401193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:13:53.537862532 +0000 UTC m=+504.576009150" watchObservedRunningTime="2026-07-03 10:13:53.539401193 +0000 UTC m=+504.577547872" Jul 03 10:13:53.558945 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.558902 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lvdb6" podStartSLOduration=2.450804426 podStartE2EDuration="25.55888686s" totalImagesPullingTime="23.108082434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:13:53.557809935 +0000 UTC m=+504.595956544" watchObservedRunningTime="2026-07-03 10:13:53.55888686 +0000 UTC m=+504.597033464" Jul 03 10:13:53.559829 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:13:53.559806 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-8qjvs\" is forbidden: User \"system:node:ip-10-0-143-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-105.ec2.internal' and this object" podUID="474caa4b-cfc3-4f0f-8973-a043928fc531" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8qjvs" Jul 03 10:13:53.582862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.582815 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq" podStartSLOduration=14.58280037 podStartE2EDuration="14.58280037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:13:53.581018243 +0000 UTC m=+504.619164850" watchObservedRunningTime="2026-07-03 10:13:53.58280037 +0000 UTC m=+504.620947030" Jul 03 10:13:53.606279 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:53.606203 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t" podStartSLOduration=11.606189673 podStartE2EDuration="11.606189673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:13:53.603780558 +0000 UTC m=+504.641927166" watchObservedRunningTime="2026-07-03 10:13:53.606189673 +0000 UTC m=+504.644336279" Jul 03 10:13:57.472666 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:57.472584 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:13:57.570628 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:13:57.570562 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:00.028007 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:00.027977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:00.162853 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:00.162814 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:04.326394 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:04.326348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:04.421323 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:04.421281 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:04.528381 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:04.528348 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vxnfn" Jul 03 10:14:04.528542 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:04.528434 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t" Jul 03 10:14:04.604726 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:04.604694 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq"] Jul 03 10:14:04.604923 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:04.604899 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq" podUID="6600a0b6-ae91-4f07-9871-7dc2447726e3" containerName="manager" containerID="cri-o://1e5921bae6ceb50e25c1f5c470bf319192834a01c110b987a28520481342ba6e" gracePeriod=10 Jul 03 10:14:04.606936 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:04.606904 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq" Jul 03 10:14:04.842900 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:04.842881 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq" Jul 03 10:14:04.930351 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:04.930283 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxqxz\" (UniqueName: \"kubernetes.io/projected/6600a0b6-ae91-4f07-9871-7dc2447726e3-kube-api-access-wxqxz\") pod \"6600a0b6-ae91-4f07-9871-7dc2447726e3\" (UID: \"6600a0b6-ae91-4f07-9871-7dc2447726e3\") " Jul 03 10:14:04.930351 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:04.930332 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6600a0b6-ae91-4f07-9871-7dc2447726e3-extensions-socket-volume\") pod \"6600a0b6-ae91-4f07-9871-7dc2447726e3\" (UID: \"6600a0b6-ae91-4f07-9871-7dc2447726e3\") " Jul 03 10:14:04.930748 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:04.930724 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6600a0b6-ae91-4f07-9871-7dc2447726e3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6600a0b6-ae91-4f07-9871-7dc2447726e3" (UID: "6600a0b6-ae91-4f07-9871-7dc2447726e3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:14:04.932166 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:04.932146 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6600a0b6-ae91-4f07-9871-7dc2447726e3-kube-api-access-wxqxz" (OuterVolumeSpecName: "kube-api-access-wxqxz") pod "6600a0b6-ae91-4f07-9871-7dc2447726e3" (UID: "6600a0b6-ae91-4f07-9871-7dc2447726e3"). InnerVolumeSpecName "kube-api-access-wxqxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 10:14:05.031914 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:05.031867 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxqxz\" (UniqueName: \"kubernetes.io/projected/6600a0b6-ae91-4f07-9871-7dc2447726e3-kube-api-access-wxqxz\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:14:05.031914 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:05.031896 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6600a0b6-ae91-4f07-9871-7dc2447726e3-extensions-socket-volume\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:14:05.576358 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:05.576320 2579 generic.go:356] "Generic (PLEG): container finished" podID="6600a0b6-ae91-4f07-9871-7dc2447726e3" containerID="1e5921bae6ceb50e25c1f5c470bf319192834a01c110b987a28520481342ba6e" exitCode=0 Jul 03 10:14:05.576358 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:05.576357 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq" Jul 03 10:14:05.576750 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:05.576403 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq" event={"ID":"6600a0b6-ae91-4f07-9871-7dc2447726e3","Type":"ContainerDied","Data":"1e5921bae6ceb50e25c1f5c470bf319192834a01c110b987a28520481342ba6e"} Jul 03 10:14:05.576750 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:05.576449 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq" event={"ID":"6600a0b6-ae91-4f07-9871-7dc2447726e3","Type":"ContainerDied","Data":"d84ca987fd4a3a58edc755d1e3f1a24a0718477f386a18e4b46fd81a2ca66fd2"} Jul 03 10:14:05.576750 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:05.576471 2579 scope.go:117] "RemoveContainer" containerID="1e5921bae6ceb50e25c1f5c470bf319192834a01c110b987a28520481342ba6e" Jul 03 10:14:05.584665 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:05.584641 2579 scope.go:117] "RemoveContainer" containerID="1e5921bae6ceb50e25c1f5c470bf319192834a01c110b987a28520481342ba6e" Jul 03 10:14:05.584957 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:14:05.584939 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5921bae6ceb50e25c1f5c470bf319192834a01c110b987a28520481342ba6e\": container with ID starting with 1e5921bae6ceb50e25c1f5c470bf319192834a01c110b987a28520481342ba6e not found: ID does not exist" containerID="1e5921bae6ceb50e25c1f5c470bf319192834a01c110b987a28520481342ba6e" Jul 03 10:14:05.585023 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:05.584968 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5921bae6ceb50e25c1f5c470bf319192834a01c110b987a28520481342ba6e"} err="failed to get container status \"1e5921bae6ceb50e25c1f5c470bf319192834a01c110b987a28520481342ba6e\": rpc error: code = NotFound desc = could not find container \"1e5921bae6ceb50e25c1f5c470bf319192834a01c110b987a28520481342ba6e\": container with ID starting with 1e5921bae6ceb50e25c1f5c470bf319192834a01c110b987a28520481342ba6e not found: ID does not exist" Jul 03 10:14:05.614087 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:05.614049 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq"] Jul 03 10:14:05.617294 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:05.617268 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vnbnq"] Jul 03 10:14:07.317544 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:07.317514 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:07.427334 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:07.427301 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:07.530808 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:07.530773 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6600a0b6-ae91-4f07-9871-7dc2447726e3" path="/var/lib/kubelet/pods/6600a0b6-ae91-4f07-9871-7dc2447726e3/volumes" Jul 03 10:14:17.115176 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:17.115142 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:17.221057 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:17.221022 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:17.665393 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:17.665362 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:17.738398 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:17.738367 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:20.664854 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:20.664819 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:20.731015 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:20.730982 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:20.957089 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:20.956987 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:21.116008 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:21.115978 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:23.053192 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.053159 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:23.154707 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.154670 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:23.253430 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.253396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:23.578811 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.578777 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:14:23.579108 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.579096 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6600a0b6-ae91-4f07-9871-7dc2447726e3" containerName="manager" Jul 03 10:14:23.579159 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.579110 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6600a0b6-ae91-4f07-9871-7dc2447726e3" containerName="manager" Jul 03 10:14:23.579193 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.579163 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6600a0b6-ae91-4f07-9871-7dc2447726e3" containerName="manager" Jul 03 10:14:23.592538 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.592512 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:14:23.592697 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.592606 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-tsqcn" Jul 03 10:14:23.594723 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.594693 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 03 10:14:23.618274 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.618239 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:14:23.688157 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.688124 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a1c36347-b59d-4225-a2a5-1772d3433cb9-config-file\") pod \"limitador-limitador-54d6d988d5-tsqcn\" (UID: \"a1c36347-b59d-4225-a2a5-1772d3433cb9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tsqcn" Jul 03 10:14:23.688357 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.688172 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d48f2\" (UniqueName: \"kubernetes.io/projected/a1c36347-b59d-4225-a2a5-1772d3433cb9-kube-api-access-d48f2\") pod \"limitador-limitador-54d6d988d5-tsqcn\" (UID: \"a1c36347-b59d-4225-a2a5-1772d3433cb9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tsqcn" Jul 03 10:14:23.788934 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.788896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a1c36347-b59d-4225-a2a5-1772d3433cb9-config-file\") pod \"limitador-limitador-54d6d988d5-tsqcn\" (UID: \"a1c36347-b59d-4225-a2a5-1772d3433cb9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tsqcn" Jul 03 10:14:23.789093 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.788949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d48f2\" (UniqueName: \"kubernetes.io/projected/a1c36347-b59d-4225-a2a5-1772d3433cb9-kube-api-access-d48f2\") pod \"limitador-limitador-54d6d988d5-tsqcn\" (UID: \"a1c36347-b59d-4225-a2a5-1772d3433cb9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tsqcn" Jul 03 10:14:23.789559 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.789536 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a1c36347-b59d-4225-a2a5-1772d3433cb9-config-file\") pod \"limitador-limitador-54d6d988d5-tsqcn\" (UID: \"a1c36347-b59d-4225-a2a5-1772d3433cb9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tsqcn" Jul 03 10:14:23.797238 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.797191 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d48f2\" (UniqueName: \"kubernetes.io/projected/a1c36347-b59d-4225-a2a5-1772d3433cb9-kube-api-access-d48f2\") pod \"limitador-limitador-54d6d988d5-tsqcn\" (UID: \"a1c36347-b59d-4225-a2a5-1772d3433cb9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tsqcn" Jul 03 10:14:23.903745 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.903715 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-tsqcn" Jul 03 10:14:23.916385 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.916352 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-cfqlf"] Jul 03 10:14:23.925584 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.925561 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cfqlf" Jul 03 10:14:23.927263 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.927235 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-cfqlf"] Jul 03 10:14:23.928017 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.927984 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-w6vxx\"" Jul 03 10:14:23.951871 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:23.951845 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:24.056610 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:24.056575 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:24.070019 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:24.069984 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:14:24.074493 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:14:24.074465 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c36347_b59d_4225_a2a5_1772d3433cb9.slice/crio-7abef942185d7ce93c85e5852cd64330fd43e0e7bf52171373369cf6b85d3478 WatchSource:0}: Error finding container 7abef942185d7ce93c85e5852cd64330fd43e0e7bf52171373369cf6b85d3478: Status 404 returned error can't find the container with id 7abef942185d7ce93c85e5852cd64330fd43e0e7bf52171373369cf6b85d3478 Jul 03 10:14:24.091268 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:24.091229 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r62x8\" (UniqueName: \"kubernetes.io/projected/3272575b-c42d-449d-8782-c2b46db8d31f-kube-api-access-r62x8\") pod \"authorino-58cd955675-cfqlf\" (UID: \"3272575b-c42d-449d-8782-c2b46db8d31f\") " pod="kuadrant-system/authorino-58cd955675-cfqlf" Jul 03 10:14:24.192080 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:24.192009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r62x8\" (UniqueName: \"kubernetes.io/projected/3272575b-c42d-449d-8782-c2b46db8d31f-kube-api-access-r62x8\") pod \"authorino-58cd955675-cfqlf\" (UID: \"3272575b-c42d-449d-8782-c2b46db8d31f\") " pod="kuadrant-system/authorino-58cd955675-cfqlf" Jul 03 10:14:24.200018 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:24.199992 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r62x8\" (UniqueName: \"kubernetes.io/projected/3272575b-c42d-449d-8782-c2b46db8d31f-kube-api-access-r62x8\") pod \"authorino-58cd955675-cfqlf\" (UID: \"3272575b-c42d-449d-8782-c2b46db8d31f\") " pod="kuadrant-system/authorino-58cd955675-cfqlf" Jul 03 10:14:24.251428 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:24.251396 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cfqlf" Jul 03 10:14:24.472185 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:24.472104 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-cfqlf"] Jul 03 10:14:24.474531 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:14:24.474503 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3272575b_c42d_449d_8782_c2b46db8d31f.slice/crio-3beb555147d1d2799f1dc2a692780f22827e1d140df2d44da05eef6160ae489f WatchSource:0}: Error finding container 3beb555147d1d2799f1dc2a692780f22827e1d140df2d44da05eef6160ae489f: Status 404 returned error can't find the container with id 3beb555147d1d2799f1dc2a692780f22827e1d140df2d44da05eef6160ae489f Jul 03 10:14:24.665226 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:24.665176 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-tsqcn" event={"ID":"a1c36347-b59d-4225-a2a5-1772d3433cb9","Type":"ContainerStarted","Data":"7abef942185d7ce93c85e5852cd64330fd43e0e7bf52171373369cf6b85d3478"} Jul 03 10:14:24.666277 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:24.666243 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cfqlf" event={"ID":"3272575b-c42d-449d-8782-c2b46db8d31f","Type":"ContainerStarted","Data":"3beb555147d1d2799f1dc2a692780f22827e1d140df2d44da05eef6160ae489f"} Jul 03 10:14:29.164517 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:29.164486 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-cfqlf"] Jul 03 10:14:29.700684 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:29.700641 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-tsqcn" event={"ID":"a1c36347-b59d-4225-a2a5-1772d3433cb9","Type":"ContainerStarted","Data":"b2a2a6f548cdb5da60c8b67775d31ebb59972853c59d09043e08693a1e01eb92"} Jul 03 10:14:29.700878 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:29.700796 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-tsqcn" Jul 03 10:14:29.702016 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:29.701994 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cfqlf" event={"ID":"3272575b-c42d-449d-8782-c2b46db8d31f","Type":"ContainerStarted","Data":"9322a09bfada20d67c93193e318a24142ab576d75cbf137a8222759ef1d91d3d"} Jul 03 10:14:29.720598 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:29.720556 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-tsqcn" podStartSLOduration=1.975071803 podStartE2EDuration="6.720543323s" totalImagesPullingTime="4.74547152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:14:29.718811854 +0000 UTC m=+540.756958456" watchObservedRunningTime="2026-07-03 10:14:29.720543323 +0000 UTC m=+540.758689932" Jul 03 10:14:29.733980 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:29.733946 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-cfqlf" podStartSLOduration=2.387323539 podStartE2EDuration="6.73393584s" totalImagesPullingTime="4.346612301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:14:29.732832766 +0000 UTC m=+540.770979374" watchObservedRunningTime="2026-07-03 10:14:29.73393584 +0000 UTC m=+540.772082447" Jul 03 10:14:30.705968 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:30.705926 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-cfqlf" podUID="3272575b-c42d-449d-8782-c2b46db8d31f" containerName="authorino" containerID="cri-o://9322a09bfada20d67c93193e318a24142ab576d75cbf137a8222759ef1d91d3d" gracePeriod=30 Jul 03 10:14:30.964614 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:30.964557 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cfqlf" Jul 03 10:14:31.058581 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:31.058545 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r62x8\" (UniqueName: \"kubernetes.io/projected/3272575b-c42d-449d-8782-c2b46db8d31f-kube-api-access-r62x8\") pod \"3272575b-c42d-449d-8782-c2b46db8d31f\" (UID: \"3272575b-c42d-449d-8782-c2b46db8d31f\") " Jul 03 10:14:31.060568 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:31.060539 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3272575b-c42d-449d-8782-c2b46db8d31f-kube-api-access-r62x8" (OuterVolumeSpecName: "kube-api-access-r62x8") pod "3272575b-c42d-449d-8782-c2b46db8d31f" (UID: "3272575b-c42d-449d-8782-c2b46db8d31f"). InnerVolumeSpecName "kube-api-access-r62x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 10:14:31.159083 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:31.159043 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r62x8\" (UniqueName: \"kubernetes.io/projected/3272575b-c42d-449d-8782-c2b46db8d31f-kube-api-access-r62x8\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:14:31.710946 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:31.710907 2579 generic.go:356] "Generic (PLEG): container finished" podID="3272575b-c42d-449d-8782-c2b46db8d31f" containerID="9322a09bfada20d67c93193e318a24142ab576d75cbf137a8222759ef1d91d3d" exitCode=0 Jul 03 10:14:31.711373 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:31.710953 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cfqlf" Jul 03 10:14:31.711373 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:31.710984 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cfqlf" event={"ID":"3272575b-c42d-449d-8782-c2b46db8d31f","Type":"ContainerDied","Data":"9322a09bfada20d67c93193e318a24142ab576d75cbf137a8222759ef1d91d3d"} Jul 03 10:14:31.711373 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:31.711031 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cfqlf" event={"ID":"3272575b-c42d-449d-8782-c2b46db8d31f","Type":"ContainerDied","Data":"3beb555147d1d2799f1dc2a692780f22827e1d140df2d44da05eef6160ae489f"} Jul 03 10:14:31.711373 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:31.711048 2579 scope.go:117] "RemoveContainer" containerID="9322a09bfada20d67c93193e318a24142ab576d75cbf137a8222759ef1d91d3d" Jul 03 10:14:31.720948 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:31.720929 2579 scope.go:117] "RemoveContainer" containerID="9322a09bfada20d67c93193e318a24142ab576d75cbf137a8222759ef1d91d3d" Jul 03 10:14:31.721227 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:14:31.721197 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9322a09bfada20d67c93193e318a24142ab576d75cbf137a8222759ef1d91d3d\": container with ID starting with 9322a09bfada20d67c93193e318a24142ab576d75cbf137a8222759ef1d91d3d not found: ID does not exist" containerID="9322a09bfada20d67c93193e318a24142ab576d75cbf137a8222759ef1d91d3d" Jul 03 10:14:31.721311 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:31.721240 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9322a09bfada20d67c93193e318a24142ab576d75cbf137a8222759ef1d91d3d"} err="failed to get container status \"9322a09bfada20d67c93193e318a24142ab576d75cbf137a8222759ef1d91d3d\": rpc error: code = NotFound desc = could not find container \"9322a09bfada20d67c93193e318a24142ab576d75cbf137a8222759ef1d91d3d\": container with ID starting with 9322a09bfada20d67c93193e318a24142ab576d75cbf137a8222759ef1d91d3d not found: ID does not exist" Jul 03 10:14:31.731134 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:31.731108 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-cfqlf"] Jul 03 10:14:31.736578 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:31.736554 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-cfqlf"] Jul 03 10:14:33.531862 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:33.531823 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3272575b-c42d-449d-8782-c2b46db8d31f" path="/var/lib/kubelet/pods/3272575b-c42d-449d-8782-c2b46db8d31f/volumes" Jul 03 10:14:40.706722 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:40.706693 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-tsqcn" Jul 03 10:14:44.625074 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:44.625036 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:57.588580 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:57.588543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:14:57.749913 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:14:57.749873 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:15:03.676616 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:15:03.676559 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:15:03.783452 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:15:03.783397 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:15:03.848278 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:15:03.848246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:15:06.744794 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:15:06.744742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:15:06.830748 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:15:06.830713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:15:41.245841 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:15:41.245803 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:15:41.646187 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:15:41.646144 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:15:42.041603 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:15:42.041510 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:15:42.139581 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:15:42.139547 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:15:42.247208 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:15:42.247179 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:15:42.330404 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:15:42.330327 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:15:52.470486 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:15:52.470396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:15:53.627791 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:15:53.627753 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:15:55.832266 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:15:55.832212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:15:59.204301 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:15:59.204262 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:15:59.603640 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:15:59.603608 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:16:01.693353 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:01.693321 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:16:04.233178 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:04.233148 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:16:05.021253 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:05.020474 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:16:17.222138 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:17.222102 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:16:18.389464 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:18.389428 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:16:20.032945 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:20.032913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:16:24.011637 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:24.011604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:16:24.792336 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:24.792294 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:16:26.995410 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:26.995368 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq"] Jul 03 10:16:26.995802 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:26.995704 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3272575b-c42d-449d-8782-c2b46db8d31f" containerName="authorino" Jul 03 10:16:26.995802 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:26.995716 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3272575b-c42d-449d-8782-c2b46db8d31f" containerName="authorino" Jul 03 10:16:26.995802 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:26.995779 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3272575b-c42d-449d-8782-c2b46db8d31f" containerName="authorino" Jul 03 10:16:26.997586 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:26.997552 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:26.999737 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:26.999716 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5mnsz\"" Jul 03 10:16:26.999867 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:26.999760 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 03 10:16:27.000000 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:26.999987 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 03 10:16:27.000369 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.000351 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 03 10:16:27.011339 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.011313 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq"] Jul 03 10:16:27.031255 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.031225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:16:27.098539 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.098499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c3f7179-b886-40fd-abd8-6dec502217e0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpdpq\" (UID: \"6c3f7179-b886-40fd-abd8-6dec502217e0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:27.098539 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.098544 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c3f7179-b886-40fd-abd8-6dec502217e0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpdpq\" (UID: \"6c3f7179-b886-40fd-abd8-6dec502217e0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:27.098800 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.098583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c3f7179-b886-40fd-abd8-6dec502217e0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpdpq\" (UID: \"6c3f7179-b886-40fd-abd8-6dec502217e0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:27.098800 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.098609 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c3f7179-b886-40fd-abd8-6dec502217e0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpdpq\" (UID: \"6c3f7179-b886-40fd-abd8-6dec502217e0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:27.098800 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.098628 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c3f7179-b886-40fd-abd8-6dec502217e0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpdpq\" (UID: \"6c3f7179-b886-40fd-abd8-6dec502217e0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:27.098800 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.098650 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5j57\" (UniqueName: \"kubernetes.io/projected/6c3f7179-b886-40fd-abd8-6dec502217e0-kube-api-access-h5j57\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpdpq\" (UID: \"6c3f7179-b886-40fd-abd8-6dec502217e0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:27.199589 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.199553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c3f7179-b886-40fd-abd8-6dec502217e0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpdpq\" (UID: \"6c3f7179-b886-40fd-abd8-6dec502217e0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:27.199589 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.199592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5j57\" (UniqueName: \"kubernetes.io/projected/6c3f7179-b886-40fd-abd8-6dec502217e0-kube-api-access-h5j57\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpdpq\" (UID: \"6c3f7179-b886-40fd-abd8-6dec502217e0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:27.199829 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.199639 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c3f7179-b886-40fd-abd8-6dec502217e0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpdpq\" (UID: \"6c3f7179-b886-40fd-abd8-6dec502217e0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:27.199829 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.199662 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c3f7179-b886-40fd-abd8-6dec502217e0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpdpq\" (UID: \"6c3f7179-b886-40fd-abd8-6dec502217e0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:27.199829 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.199695 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c3f7179-b886-40fd-abd8-6dec502217e0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpdpq\" (UID: \"6c3f7179-b886-40fd-abd8-6dec502217e0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:27.199992 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.199903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c3f7179-b886-40fd-abd8-6dec502217e0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpdpq\" (UID: \"6c3f7179-b886-40fd-abd8-6dec502217e0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:27.199992 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.199964 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c3f7179-b886-40fd-abd8-6dec502217e0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpdpq\" (UID: \"6c3f7179-b886-40fd-abd8-6dec502217e0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:27.200134 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.200113 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c3f7179-b886-40fd-abd8-6dec502217e0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpdpq\" (UID: \"6c3f7179-b886-40fd-abd8-6dec502217e0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:27.200209 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.200192 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c3f7179-b886-40fd-abd8-6dec502217e0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpdpq\" (UID: \"6c3f7179-b886-40fd-abd8-6dec502217e0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:27.202013 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.201993 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c3f7179-b886-40fd-abd8-6dec502217e0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpdpq\" (UID: \"6c3f7179-b886-40fd-abd8-6dec502217e0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:27.202192 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.202176 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c3f7179-b886-40fd-abd8-6dec502217e0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpdpq\" (UID: \"6c3f7179-b886-40fd-abd8-6dec502217e0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:27.207820 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.207800 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5j57\" (UniqueName: \"kubernetes.io/projected/6c3f7179-b886-40fd-abd8-6dec502217e0-kube-api-access-h5j57\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bpdpq\" (UID: \"6c3f7179-b886-40fd-abd8-6dec502217e0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:27.314520 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.314428 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:27.446959 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.446934 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq"] Jul 03 10:16:27.449084 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:16:27.449053 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3f7179_b886_40fd_abd8_6dec502217e0.slice/crio-b0d9c2211cb6258c890115edf0a2d204c0b6044e6f487450451c0dd0b7390348 WatchSource:0}: Error finding container b0d9c2211cb6258c890115edf0a2d204c0b6044e6f487450451c0dd0b7390348: Status 404 returned error can't find the container with id b0d9c2211cb6258c890115edf0a2d204c0b6044e6f487450451c0dd0b7390348 Jul 03 10:16:27.450485 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:27.450467 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 03 10:16:28.223280 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:28.223241 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" event={"ID":"6c3f7179-b886-40fd-abd8-6dec502217e0","Type":"ContainerStarted","Data":"b0d9c2211cb6258c890115edf0a2d204c0b6044e6f487450451c0dd0b7390348"} Jul 03 10:16:29.192886 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:29.192854 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:16:29.230511 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:29.230480 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" event={"ID":"6c3f7179-b886-40fd-abd8-6dec502217e0","Type":"ContainerStarted","Data":"f787fc1f5532854cc9c18273c7f7cadb1ec9977974a355dd1d25b2b8459cbd55"} Jul 03 10:16:29.253394 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:29.253346 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" podStartSLOduration=2.184480493 podStartE2EDuration="3.253330703s" totalImagesPullingTime="1.06885021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:16:29.252111289 +0000 UTC m=+660.290257897" watchObservedRunningTime="2026-07-03 10:16:29.253330703 +0000 UTC m=+660.291477310" Jul 03 10:16:37.315567 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:37.315531 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:37.315983 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:37.315595 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:37.327885 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:37.327854 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:38.281663 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:38.281629 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bpdpq" Jul 03 10:16:50.799633 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:50.799596 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv"] Jul 03 10:16:50.804622 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:50.804606 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:50.806987 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:50.806971 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 03 10:16:50.811538 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:50.811313 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv"] Jul 03 10:16:50.900084 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:50.900049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5876fe42-f276-4325-91c6-f6a60ed620a9-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv\" (UID: \"5876fe42-f276-4325-91c6-f6a60ed620a9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:50.900084 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:50.900085 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5876fe42-f276-4325-91c6-f6a60ed620a9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv\" (UID: \"5876fe42-f276-4325-91c6-f6a60ed620a9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:50.900318 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:50.900156 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5876fe42-f276-4325-91c6-f6a60ed620a9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv\" (UID: \"5876fe42-f276-4325-91c6-f6a60ed620a9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:50.900318 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:50.900190 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5876fe42-f276-4325-91c6-f6a60ed620a9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv\" (UID: \"5876fe42-f276-4325-91c6-f6a60ed620a9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:50.900318 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:50.900280 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87klm\" (UniqueName: \"kubernetes.io/projected/5876fe42-f276-4325-91c6-f6a60ed620a9-kube-api-access-87klm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv\" (UID: \"5876fe42-f276-4325-91c6-f6a60ed620a9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:50.900318 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:50.900301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5876fe42-f276-4325-91c6-f6a60ed620a9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv\" (UID: \"5876fe42-f276-4325-91c6-f6a60ed620a9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:51.001141 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:51.001111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87klm\" (UniqueName: \"kubernetes.io/projected/5876fe42-f276-4325-91c6-f6a60ed620a9-kube-api-access-87klm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv\" (UID: \"5876fe42-f276-4325-91c6-f6a60ed620a9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:51.001141 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:51.001145 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5876fe42-f276-4325-91c6-f6a60ed620a9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv\" (UID: \"5876fe42-f276-4325-91c6-f6a60ed620a9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:51.001409 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:51.001170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5876fe42-f276-4325-91c6-f6a60ed620a9-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv\" (UID: \"5876fe42-f276-4325-91c6-f6a60ed620a9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:51.001409 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:51.001197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5876fe42-f276-4325-91c6-f6a60ed620a9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv\" (UID: \"5876fe42-f276-4325-91c6-f6a60ed620a9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:51.001409 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:51.001306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5876fe42-f276-4325-91c6-f6a60ed620a9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv\" (UID: \"5876fe42-f276-4325-91c6-f6a60ed620a9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:51.001409 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:51.001333 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5876fe42-f276-4325-91c6-f6a60ed620a9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv\" (UID: \"5876fe42-f276-4325-91c6-f6a60ed620a9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:51.001673 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:51.001643 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5876fe42-f276-4325-91c6-f6a60ed620a9-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv\" (UID: \"5876fe42-f276-4325-91c6-f6a60ed620a9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:51.001747 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:51.001668 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5876fe42-f276-4325-91c6-f6a60ed620a9-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv\" (UID: \"5876fe42-f276-4325-91c6-f6a60ed620a9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:51.001747 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:51.001724 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5876fe42-f276-4325-91c6-f6a60ed620a9-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv\" (UID: \"5876fe42-f276-4325-91c6-f6a60ed620a9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:51.003452 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:51.003431 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5876fe42-f276-4325-91c6-f6a60ed620a9-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv\" (UID: \"5876fe42-f276-4325-91c6-f6a60ed620a9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:51.003760 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:51.003742 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5876fe42-f276-4325-91c6-f6a60ed620a9-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv\" (UID: \"5876fe42-f276-4325-91c6-f6a60ed620a9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:51.008924 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:51.008898 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87klm\" (UniqueName: \"kubernetes.io/projected/5876fe42-f276-4325-91c6-f6a60ed620a9-kube-api-access-87klm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv\" (UID: \"5876fe42-f276-4325-91c6-f6a60ed620a9\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:51.043990 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:51.043958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:16:51.114083 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:51.114050 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:16:51.244370 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:51.244337 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv"] Jul 03 10:16:51.247908 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:16:51.247886 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5876fe42_f276_4325_91c6_f6a60ed620a9.slice/crio-f309e4f30f1c1e4a945fa088f60e4533f14785bfcc00aac21a209ccf05e23556 WatchSource:0}: Error finding container f309e4f30f1c1e4a945fa088f60e4533f14785bfcc00aac21a209ccf05e23556: Status 404 returned error can't find the container with id f309e4f30f1c1e4a945fa088f60e4533f14785bfcc00aac21a209ccf05e23556 Jul 03 10:16:51.324662 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:51.324627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" event={"ID":"5876fe42-f276-4325-91c6-f6a60ed620a9","Type":"ContainerStarted","Data":"f309e4f30f1c1e4a945fa088f60e4533f14785bfcc00aac21a209ccf05e23556"} Jul 03 10:16:52.331714 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:52.331675 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" event={"ID":"5876fe42-f276-4325-91c6-f6a60ed620a9","Type":"ContainerStarted","Data":"09cd96585689eab76c940a98b3ca39d8287f8615e91339f9bae64ecaf85a4b03"} Jul 03 10:16:52.350054 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:52.350003 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" podStartSLOduration=2.184751341 podStartE2EDuration="2.349987185s" totalImagesPullingTime="165.235844ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:16:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:16:52.347903015 +0000 UTC m=+683.386049635" watchObservedRunningTime="2026-07-03 10:16:52.349987185 +0000 UTC m=+683.388133823" Jul 03 10:16:52.691876 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:16:52.691838 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:17:01.115187 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:17:01.115146 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:17:01.115187 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:17:01.115200 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:17:01.127753 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:17:01.127729 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:17:01.393751 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:17:01.393666 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv" Jul 03 10:17:17.396709 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:17:17.396668 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:18:13.259694 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:13.259663 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6bcfbc4d69-ldgkh"] Jul 03 10:18:13.262824 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:13.262802 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bcfbc4d69-ldgkh" Jul 03 10:18:13.265874 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:13.265853 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 03 10:18:13.266954 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:13.266938 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-w6vxx\"" Jul 03 10:18:13.274988 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:13.274969 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bcfbc4d69-ldgkh"] Jul 03 10:18:13.316884 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:13.316851 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnpzc\" (UniqueName: \"kubernetes.io/projected/03edaf58-b0b1-4aec-b60a-1cf296dcc1cd-kube-api-access-dnpzc\") pod \"authorino-6bcfbc4d69-ldgkh\" (UID: \"03edaf58-b0b1-4aec-b60a-1cf296dcc1cd\") " pod="kuadrant-system/authorino-6bcfbc4d69-ldgkh" Jul 03 10:18:13.317047 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:13.316902 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/03edaf58-b0b1-4aec-b60a-1cf296dcc1cd-tls-cert\") pod \"authorino-6bcfbc4d69-ldgkh\" (UID: \"03edaf58-b0b1-4aec-b60a-1cf296dcc1cd\") " pod="kuadrant-system/authorino-6bcfbc4d69-ldgkh" Jul 03 10:18:13.418211 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:13.418174 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnpzc\" (UniqueName: \"kubernetes.io/projected/03edaf58-b0b1-4aec-b60a-1cf296dcc1cd-kube-api-access-dnpzc\") pod \"authorino-6bcfbc4d69-ldgkh\" (UID: \"03edaf58-b0b1-4aec-b60a-1cf296dcc1cd\") " pod="kuadrant-system/authorino-6bcfbc4d69-ldgkh" Jul 03 10:18:13.418444 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:13.418260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/03edaf58-b0b1-4aec-b60a-1cf296dcc1cd-tls-cert\") pod \"authorino-6bcfbc4d69-ldgkh\" (UID: \"03edaf58-b0b1-4aec-b60a-1cf296dcc1cd\") " pod="kuadrant-system/authorino-6bcfbc4d69-ldgkh" Jul 03 10:18:13.420630 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:13.420600 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/03edaf58-b0b1-4aec-b60a-1cf296dcc1cd-tls-cert\") pod \"authorino-6bcfbc4d69-ldgkh\" (UID: \"03edaf58-b0b1-4aec-b60a-1cf296dcc1cd\") " pod="kuadrant-system/authorino-6bcfbc4d69-ldgkh" Jul 03 10:18:13.429533 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:13.429509 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnpzc\" (UniqueName: \"kubernetes.io/projected/03edaf58-b0b1-4aec-b60a-1cf296dcc1cd-kube-api-access-dnpzc\") pod \"authorino-6bcfbc4d69-ldgkh\" (UID: \"03edaf58-b0b1-4aec-b60a-1cf296dcc1cd\") " pod="kuadrant-system/authorino-6bcfbc4d69-ldgkh" Jul 03 10:18:13.571674 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:13.571587 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bcfbc4d69-ldgkh" Jul 03 10:18:13.643246 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:13.643174 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:18:13.746883 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:13.746859 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bcfbc4d69-ldgkh"] Jul 03 10:18:13.748508 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:18:13.748479 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03edaf58_b0b1_4aec_b60a_1cf296dcc1cd.slice/crio-c40882291d5f2f68153603c4c1f6f24420dc82123c13612946ec68ea505ca94b WatchSource:0}: Error finding container c40882291d5f2f68153603c4c1f6f24420dc82123c13612946ec68ea505ca94b: Status 404 returned error can't find the container with id c40882291d5f2f68153603c4c1f6f24420dc82123c13612946ec68ea505ca94b Jul 03 10:18:13.794936 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:13.794902 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:18:14.704584 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:14.704541 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bcfbc4d69-ldgkh" event={"ID":"03edaf58-b0b1-4aec-b60a-1cf296dcc1cd","Type":"ContainerStarted","Data":"b33a2a68815ca116ffbee8ec8aebcbaeda7a815a0077f793023614ad6d6d1ff7"} Jul 03 10:18:14.705065 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:14.704599 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bcfbc4d69-ldgkh" event={"ID":"03edaf58-b0b1-4aec-b60a-1cf296dcc1cd","Type":"ContainerStarted","Data":"c40882291d5f2f68153603c4c1f6f24420dc82123c13612946ec68ea505ca94b"} Jul 03 10:18:14.762905 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:14.762848 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6bcfbc4d69-ldgkh" podStartSLOduration=1.249620455 podStartE2EDuration="1.762831756s" totalImagesPullingTime="513.211301ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:18:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:18:14.761801922 +0000 UTC m=+765.799948534" watchObservedRunningTime="2026-07-03 10:18:14.762831756 +0000 UTC m=+765.800978364" Jul 03 10:18:14.921349 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:14.921308 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:18:15.446908 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:15.446872 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:18:42.405201 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:42.405159 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:18:52.590725 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:52.590648 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:18:55.313782 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:55.313742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:18:55.442900 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:18:55.442861 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:19:04.614284 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:19:04.614248 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:19:13.303847 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:19:13.303812 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:19:24.535797 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:19:24.535759 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:19:26.101638 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:19:26.101586 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:19:27.389406 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:19:27.389373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:19:49.711261 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:19:49.711229 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:19:59.104184 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:19:59.104147 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:20:03.146414 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:20:03.146377 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:20:03.254944 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:20:03.254913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:20:03.343190 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:20:03.343160 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:20:10.184092 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:20:10.184041 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:20:10.312202 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:20:10.312166 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:20:10.430636 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:20:10.430598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:20:10.560566 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:20:10.560478 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:20:17.526983 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:20:17.526947 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:20:17.622106 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:20:17.622077 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:20:34.604773 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:20:34.604734 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:20:42.452332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:20:42.452294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:20:42.527692 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:20:42.527664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:20:42.604202 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:20:42.604164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:20:44.229211 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:20:44.229174 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:20:52.815446 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:20:52.815411 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:21:04.111856 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:21:04.111822 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:21:15.907417 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:21:15.907376 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:21:25.230875 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:21:25.230837 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:21:36.437538 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:21:36.437502 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:21:45.928464 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:21:45.928422 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:21:58.921733 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:21:58.921647 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:22:08.915739 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:22:08.915696 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:22:19.402425 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:22:19.402387 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:22:22.930359 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:22:22.930326 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:22:43.331267 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:22:43.331229 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:22:54.123314 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:22:54.123283 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:23:10.594247 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:23:10.594195 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:23:20.050085 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:23:20.050047 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:23:30.207847 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:23:30.207767 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:23:39.209190 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:23:39.209154 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:23:47.611653 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:23:47.611621 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:24:13.201466 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:13.201427 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:24:17.371410 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:17.371375 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t"] Jul 03 10:24:17.371929 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:17.371696 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t" podUID="3bb2a19f-d59f-41e0-96e6-f5a90b6e236f" containerName="manager" containerID="cri-o://e203f8fdb51ac94635983e8f1dcbbaf7cb692fc8fe8fba4aa30ee7524bc81393" gracePeriod=10 Jul 03 10:24:18.317460 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:18.317436 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t" Jul 03 10:24:18.382746 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:18.382712 2579 generic.go:356] "Generic (PLEG): container finished" podID="3bb2a19f-d59f-41e0-96e6-f5a90b6e236f" containerID="e203f8fdb51ac94635983e8f1dcbbaf7cb692fc8fe8fba4aa30ee7524bc81393" exitCode=0 Jul 03 10:24:18.383124 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:18.382754 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t" Jul 03 10:24:18.383124 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:18.382792 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t" event={"ID":"3bb2a19f-d59f-41e0-96e6-f5a90b6e236f","Type":"ContainerDied","Data":"e203f8fdb51ac94635983e8f1dcbbaf7cb692fc8fe8fba4aa30ee7524bc81393"} Jul 03 10:24:18.383124 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:18.382845 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t" event={"ID":"3bb2a19f-d59f-41e0-96e6-f5a90b6e236f","Type":"ContainerDied","Data":"456f104da8a9d2553ddb31ede0cfe297ecab0925af6e2330c76fc4f0af4b45ca"} Jul 03 10:24:18.383124 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:18.382866 2579 scope.go:117] "RemoveContainer" containerID="e203f8fdb51ac94635983e8f1dcbbaf7cb692fc8fe8fba4aa30ee7524bc81393" Jul 03 10:24:18.391569 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:18.391552 2579 scope.go:117] "RemoveContainer" containerID="e203f8fdb51ac94635983e8f1dcbbaf7cb692fc8fe8fba4aa30ee7524bc81393" Jul 03 10:24:18.391814 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:24:18.391800 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e203f8fdb51ac94635983e8f1dcbbaf7cb692fc8fe8fba4aa30ee7524bc81393\": container with ID starting with e203f8fdb51ac94635983e8f1dcbbaf7cb692fc8fe8fba4aa30ee7524bc81393 not found: ID does not exist" containerID="e203f8fdb51ac94635983e8f1dcbbaf7cb692fc8fe8fba4aa30ee7524bc81393" Jul 03 10:24:18.391858 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:18.391823 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e203f8fdb51ac94635983e8f1dcbbaf7cb692fc8fe8fba4aa30ee7524bc81393"} err="failed to get container status \"e203f8fdb51ac94635983e8f1dcbbaf7cb692fc8fe8fba4aa30ee7524bc81393\": rpc error: code = NotFound desc = could not find container \"e203f8fdb51ac94635983e8f1dcbbaf7cb692fc8fe8fba4aa30ee7524bc81393\": container with ID starting with e203f8fdb51ac94635983e8f1dcbbaf7cb692fc8fe8fba4aa30ee7524bc81393 not found: ID does not exist" Jul 03 10:24:18.452213 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:18.452132 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nrph\" (UniqueName: \"kubernetes.io/projected/3bb2a19f-d59f-41e0-96e6-f5a90b6e236f-kube-api-access-5nrph\") pod \"3bb2a19f-d59f-41e0-96e6-f5a90b6e236f\" (UID: \"3bb2a19f-d59f-41e0-96e6-f5a90b6e236f\") " Jul 03 10:24:18.452213 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:18.452195 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3bb2a19f-d59f-41e0-96e6-f5a90b6e236f-extensions-socket-volume\") pod \"3bb2a19f-d59f-41e0-96e6-f5a90b6e236f\" (UID: \"3bb2a19f-d59f-41e0-96e6-f5a90b6e236f\") " Jul 03 10:24:18.452575 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:18.452551 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb2a19f-d59f-41e0-96e6-f5a90b6e236f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3bb2a19f-d59f-41e0-96e6-f5a90b6e236f" (UID: "3bb2a19f-d59f-41e0-96e6-f5a90b6e236f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:24:18.454225 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:18.454198 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb2a19f-d59f-41e0-96e6-f5a90b6e236f-kube-api-access-5nrph" (OuterVolumeSpecName: "kube-api-access-5nrph") pod "3bb2a19f-d59f-41e0-96e6-f5a90b6e236f" (UID: "3bb2a19f-d59f-41e0-96e6-f5a90b6e236f"). InnerVolumeSpecName "kube-api-access-5nrph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 10:24:18.553305 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:18.553265 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nrph\" (UniqueName: \"kubernetes.io/projected/3bb2a19f-d59f-41e0-96e6-f5a90b6e236f-kube-api-access-5nrph\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:24:18.553305 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:18.553297 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3bb2a19f-d59f-41e0-96e6-f5a90b6e236f-extensions-socket-volume\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:24:18.704548 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:18.704483 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t"] Jul 03 10:24:18.708637 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:18.708612 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-nkw2t"] Jul 03 10:24:19.530780 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:19.530748 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bb2a19f-d59f-41e0-96e6-f5a90b6e236f" path="/var/lib/kubelet/pods/3bb2a19f-d59f-41e0-96e6-f5a90b6e236f/volumes" Jul 03 10:24:33.277039 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:24:33.276981 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:25:18.499362 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:18.499281 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzjdz"] Jul 03 10:25:18.499795 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:18.499632 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bb2a19f-d59f-41e0-96e6-f5a90b6e236f" containerName="manager" Jul 03 10:25:18.499795 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:18.499646 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bb2a19f-d59f-41e0-96e6-f5a90b6e236f" containerName="manager" Jul 03 10:25:18.499795 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:18.499707 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bb2a19f-d59f-41e0-96e6-f5a90b6e236f" containerName="manager" Jul 03 10:25:18.502437 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:18.502420 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzjdz" Jul 03 10:25:18.504985 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:18.504960 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-mds8w\"" Jul 03 10:25:18.519275 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:18.519243 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzjdz"] Jul 03 10:25:18.628975 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:18.628936 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9szt\" (UniqueName: \"kubernetes.io/projected/b3a6d5e0-cc4f-44e8-8b82-85b55c9c5301-kube-api-access-w9szt\") pod \"kuadrant-operator-controller-manager-86878c769f-wzjdz\" (UID: \"b3a6d5e0-cc4f-44e8-8b82-85b55c9c5301\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzjdz" Jul 03 10:25:18.629142 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:18.629030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d5e0-cc4f-44e8-8b82-85b55c9c5301-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wzjdz\" (UID: \"b3a6d5e0-cc4f-44e8-8b82-85b55c9c5301\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzjdz" Jul 03 10:25:18.730051 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:18.729995 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9szt\" (UniqueName: \"kubernetes.io/projected/b3a6d5e0-cc4f-44e8-8b82-85b55c9c5301-kube-api-access-w9szt\") pod \"kuadrant-operator-controller-manager-86878c769f-wzjdz\" (UID: \"b3a6d5e0-cc4f-44e8-8b82-85b55c9c5301\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzjdz" Jul 03 10:25:18.730266 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:18.730082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d5e0-cc4f-44e8-8b82-85b55c9c5301-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wzjdz\" (UID: \"b3a6d5e0-cc4f-44e8-8b82-85b55c9c5301\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzjdz" Jul 03 10:25:18.730457 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:18.730440 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3a6d5e0-cc4f-44e8-8b82-85b55c9c5301-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wzjdz\" (UID: \"b3a6d5e0-cc4f-44e8-8b82-85b55c9c5301\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzjdz" Jul 03 10:25:18.738990 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:18.738967 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9szt\" (UniqueName: \"kubernetes.io/projected/b3a6d5e0-cc4f-44e8-8b82-85b55c9c5301-kube-api-access-w9szt\") pod \"kuadrant-operator-controller-manager-86878c769f-wzjdz\" (UID: \"b3a6d5e0-cc4f-44e8-8b82-85b55c9c5301\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzjdz" Jul 03 10:25:18.813800 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:18.813710 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzjdz" Jul 03 10:25:18.952167 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:18.952130 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzjdz"] Jul 03 10:25:18.956045 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:25:18.956016 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a6d5e0_cc4f_44e8_8b82_85b55c9c5301.slice/crio-fb8335c33f0f9b306377b49381e01cc69fa1d8d2c56a901228287b3d6644eef8 WatchSource:0}: Error finding container fb8335c33f0f9b306377b49381e01cc69fa1d8d2c56a901228287b3d6644eef8: Status 404 returned error can't find the container with id fb8335c33f0f9b306377b49381e01cc69fa1d8d2c56a901228287b3d6644eef8 Jul 03 10:25:19.656427 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:19.656391 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzjdz" event={"ID":"b3a6d5e0-cc4f-44e8-8b82-85b55c9c5301","Type":"ContainerStarted","Data":"7525a7e900913fe672656d945b9974676651791a706acac876cb236268c90b55"} Jul 03 10:25:19.656788 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:19.656439 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzjdz" event={"ID":"b3a6d5e0-cc4f-44e8-8b82-85b55c9c5301","Type":"ContainerStarted","Data":"fb8335c33f0f9b306377b49381e01cc69fa1d8d2c56a901228287b3d6644eef8"} Jul 03 10:25:19.656788 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:19.656485 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzjdz" Jul 03 10:25:19.686896 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:19.686843 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzjdz" podStartSLOduration=1.6868284930000002 podStartE2EDuration="1.686828493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:25:19.685066342 +0000 UTC m=+1190.723212948" watchObservedRunningTime="2026-07-03 10:25:19.686828493 +0000 UTC m=+1190.724975100" Jul 03 10:25:30.664823 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:30.664785 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wzjdz" Jul 03 10:25:46.806673 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:25:46.806643 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:26:02.518666 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:26:02.518627 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:26:11.996080 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:26:11.996037 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:26:21.016505 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:26:21.016427 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:26:30.796606 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:26:30.796570 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:26:41.295619 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:26:41.295582 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:26:51.700062 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:26:51.700020 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:27:01.000547 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:27:01.000508 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:27:11.398378 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:27:11.398344 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:27:24.098616 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:27:24.098579 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:27:31.584583 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:27:31.584548 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:27:49.209702 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:27:49.209666 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:27:58.020349 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:27:58.020268 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:28:10.090156 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:28:10.090118 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:28:21.398987 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:28:21.398948 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:28:38.402153 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:28:38.402115 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:28:52.391876 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:28:52.391837 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:29:01.321810 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:29:01.321774 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:29:14.900053 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:29:14.900014 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:29:24.106892 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:29:24.106809 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:29:37.801842 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:29:37.801808 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:29:47.598762 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:29:47.598724 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:29:56.748969 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:29:56.748935 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:30:09.113326 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:30:09.113289 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:30:20.900011 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:30:20.899966 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:30:29.458204 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:30:29.458168 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:30:32.668501 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:30:32.668466 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:30:41.498779 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:30:41.498740 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:30:54.107397 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:30:54.107318 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:31:07.008483 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:07.008434 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:31:08.086776 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:08.086724 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:12.304601 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:12.304562 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:12.419047 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:12.419001 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:12.750051 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:12.750001 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:18.521126 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:18.521081 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:18.687312 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:18.687261 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:18.960253 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:18.960196 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:19.169482 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:19.169454 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:20.098932 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:20.098896 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:20.202361 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:20.202328 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:20.304353 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:20.304314 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:20.481307 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:20.481189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:20.848427 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:20.848394 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:20.951431 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:20.951399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:22.595198 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:22.591732 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:31:22.752833 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:22.752796 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:25.089720 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:25.089685 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:25.234589 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:25.234556 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:25.348631 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:25.348599 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:25.484964 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:25.484928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:26.938955 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:26.938928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:30.598313 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:30.598275 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:31:31.373316 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:31.373283 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:31.475343 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:31.475306 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:31.632934 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:31.632847 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:31.731411 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:31.731368 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:32.833973 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:32.833943 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:32.958288 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:32.958259 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:33.096970 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:33.096938 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:33.269717 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:33.269689 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:33.773120 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:33.773089 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:33.901259 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:33.901205 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:34.762201 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:34.762163 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:38.036021 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:38.035988 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:38.383986 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:38.383950 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:44.313821 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:44.313787 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:44.377966 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:44.377930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:50.541287 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:50.541249 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:50.642663 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:50.642627 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:50.816664 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:50.816579 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:51.073746 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:51.073656 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:51.153825 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:51.153798 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:51.998400 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:51.998372 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:52.168382 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:52.168346 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:52.272675 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:52.272589 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:52.616545 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:52.616516 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:52.784612 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:52.784580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:52.884419 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:52.884345 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:54.407778 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:54.407741 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:31:54.478019 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:54.477982 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:56.435546 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:56.435514 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:56.600829 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:56.600793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:56.722784 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:56.722699 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:56.927646 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:56.927617 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:31:57.976620 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:31:57.976592 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:32:02.793073 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:32:02.793035 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:32:03.019109 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:32:03.019079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:32:03.165672 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:32:03.165633 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:32:03.572283 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:32:03.572191 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:32:03.814554 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:32:03.814524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:32:05.220446 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:32:05.220415 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:32:11.293174 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:32:11.293130 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:32:11.411946 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:32:11.411899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:32:11.496917 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:32:11.496883 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:32:11.631372 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:32:11.631336 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:32:12.005188 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:32:12.005101 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:32:37.979926 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:32:37.979887 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:32:43.949064 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:32:43.949031 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:32:44.078185 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:32:44.078154 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:32:44.182613 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:32:44.182572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:32:58.820627 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:32:58.820585 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:00.006789 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:00.006758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:06.182932 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:06.182881 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:06.259544 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:06.259513 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:06.366806 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:06.366777 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:06.464339 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:06.464251 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:16.926791 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:16.926759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:17.704614 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:17.704584 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:23.862645 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:23.862602 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:23.990801 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:23.990758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:24.101000 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:24.100970 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:24.915531 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:24.915495 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:25.014747 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:25.014715 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:26.661370 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:26.661338 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:27.389639 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:27.389605 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:33:32.924662 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:32.924629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:33.019955 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:33.019912 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:33.615956 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:33.615919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:36.307405 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:36.307367 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:33:39.575678 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:39.575645 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:39.707348 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:39.707313 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:39.864096 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:39.864062 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:40.259051 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:40.258963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:40.394076 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:40.394045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:43.156962 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:43.156927 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:43.258566 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:43.258538 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:43.427380 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:43.427291 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:43.572437 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:43.572406 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:44.134804 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:44.134765 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:44.243632 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:44.243604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:47.642742 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:47.642710 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:48.096540 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:48.096495 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:33:48.336923 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:48.336887 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:48.428022 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:48.427917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:48.743834 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:48.743752 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:54.511715 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:54.511629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:54.610578 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:54.610545 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:55.062229 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:55.062181 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:55.460040 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:55.460008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:55.527337 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:55.527298 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:33:58.305041 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:33:58.305004 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:34:01.759399 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:01.759368 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:01.858714 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:01.858682 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:02.054889 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:02.054811 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:02.246131 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:02.246090 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:02.366693 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:02.366661 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:02.461546 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:02.461514 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:02.561701 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:02.561668 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:08.490332 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:08.490301 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:08.592799 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:08.592760 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:08.720274 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:08.720247 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:08.852280 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:08.852251 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:14.825975 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:14.825938 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:14.946088 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:14.946053 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:21.256194 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:21.256165 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:21.689107 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:21.689071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:22.214790 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:22.214745 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:24.491155 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:24.491116 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:24.603322 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:24.603286 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:24.742333 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:24.742250 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:24.934399 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:24.934360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:25.271357 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:25.271325 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:25.369986 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:25.369954 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:27.307076 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:27.307039 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:34:28.870834 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:28.870805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:33.746359 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:33.746323 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:37.900788 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:37.900747 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:34:49.301394 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:49.301361 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:34:49.898615 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:49.898586 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:49.998947 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:49.998910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:56.096057 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:56.096019 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:56.569434 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:56.569398 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:56.670379 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:56.670343 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:34:56.849072 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:34:56.849038 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:35:00.003562 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:35:00.003524 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:35:00.007466 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:35:00.007442 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:35:00.103232 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:35:00.103194 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:35:02.156972 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:35:02.156942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:35:07.222464 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:35:07.222432 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:35:11.793798 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:35:11.793758 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:35:44.450329 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:35:44.450286 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:35:45.643235 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:35:45.643195 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:35:47.298938 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:35:47.298896 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:35:50.378580 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:35:50.378549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:35:56.197987 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:35:56.197948 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:35:56.269401 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:35:56.269372 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:35:56.493853 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:35:56.493774 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:35:56.722541 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:35:56.722508 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:35:57.803610 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:35:57.803571 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:35:57.902061 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:35:57.902030 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:35:58.003042 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:35:58.003010 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:35:58.750695 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:35:58.750653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:35:58.848777 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:35:58.848742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:00.128946 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:00.128911 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:36:00.315501 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:00.315468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:06.280833 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:06.280802 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:06.382766 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:06.382734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:06.958976 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:06.958940 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:13.122633 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:13.122601 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:13.233983 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:13.233951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:13.388234 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:13.388136 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:14.497151 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:14.497118 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:36:23.832944 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:23.832910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:24.190907 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:24.190874 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:45.168291 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:45.168252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:51.325374 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:51.325289 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:51.430411 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:51.430377 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:51.713778 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:51.713737 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:52.071018 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:52.070939 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:52.646276 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:52.646244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:55.256538 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:55.256509 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:55.407119 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:55.407089 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:55.530452 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:55.530359 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:56.197761 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:56.197728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:56.293982 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:56.293950 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:56.423957 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:56.423907 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:36:59.741911 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:36:59.741879 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:37:00.092276 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:00.092178 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:37:00.700925 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:00.700883 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:37:01.091057 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:01.090977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:37:02.712699 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:02.712662 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:37:03.338526 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:03.338479 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:37:03.550142 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:03.550110 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:37:03.692945 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:03.692912 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:37:04.539452 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:04.539418 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:37:04.745452 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:04.745418 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:37:07.934196 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:07.934163 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:37:13.458422 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:13.458387 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:37:17.251264 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:17.251213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:37:17.384053 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:17.384024 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:37:18.316885 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.316846 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg"] Jul 03 10:37:18.320255 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.320238 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.323162 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.323137 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1d628f\"/\"default-dockercfg-prx48\"" Jul 03 10:37:18.323388 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.323365 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1d628f\"/\"openshift-service-ca.crt\"" Jul 03 10:37:18.323463 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.323401 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1d628f\"/\"kube-root-ca.crt\"" Jul 03 10:37:18.323463 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.323413 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1d628f\"/\"test-model-1d628f-kserve-self-signed-certs\"" Jul 03 10:37:18.335848 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.335827 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg"] Jul 03 10:37:18.416268 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.416237 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-model-cache\") pod \"test-model-1d628f-kserve-5d7449cfd7-svllg\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.416422 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.416275 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jvwh\" (UniqueName: \"kubernetes.io/projected/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-kube-api-access-4jvwh\") pod \"test-model-1d628f-kserve-5d7449cfd7-svllg\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.416422 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.416312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-tls-certs\") pod \"test-model-1d628f-kserve-5d7449cfd7-svllg\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.416422 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.416334 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-home\") pod \"test-model-1d628f-kserve-5d7449cfd7-svllg\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.416552 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.416431 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-tmp-dir\") pod \"test-model-1d628f-kserve-5d7449cfd7-svllg\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.416552 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.416460 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-dshm\") pod \"test-model-1d628f-kserve-5d7449cfd7-svllg\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.433137 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.433097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:37:18.517722 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.517686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-model-cache\") pod \"test-model-1d628f-kserve-5d7449cfd7-svllg\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.517722 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.517726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jvwh\" (UniqueName: \"kubernetes.io/projected/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-kube-api-access-4jvwh\") pod \"test-model-1d628f-kserve-5d7449cfd7-svllg\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.517939 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.517763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-tls-certs\") pod \"test-model-1d628f-kserve-5d7449cfd7-svllg\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.517939 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.517790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-home\") pod \"test-model-1d628f-kserve-5d7449cfd7-svllg\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.517939 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.517860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-tmp-dir\") pod \"test-model-1d628f-kserve-5d7449cfd7-svllg\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.517939 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.517885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-dshm\") pod \"test-model-1d628f-kserve-5d7449cfd7-svllg\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.518387 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.518359 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-home\") pod \"test-model-1d628f-kserve-5d7449cfd7-svllg\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.518504 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.518372 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-model-cache\") pod \"test-model-1d628f-kserve-5d7449cfd7-svllg\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.518504 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.518440 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-tmp-dir\") pod \"test-model-1d628f-kserve-5d7449cfd7-svllg\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.520162 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.520138 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-dshm\") pod \"test-model-1d628f-kserve-5d7449cfd7-svllg\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.520497 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.520479 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-tls-certs\") pod \"test-model-1d628f-kserve-5d7449cfd7-svllg\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.526776 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.526755 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jvwh\" (UniqueName: \"kubernetes.io/projected/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-kube-api-access-4jvwh\") pod \"test-model-1d628f-kserve-5d7449cfd7-svllg\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.629195 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.629158 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:18.775804 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.775771 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg"] Jul 03 10:37:18.778943 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:37:18.778907 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b4d9aa9_cfe0_4fd0_9908_cfeee0946807.slice/crio-dbc80c37910d3e7a38c6e2bb9052058a511ab0d9c57241adff48d66c8f91f54d WatchSource:0}: Error finding container dbc80c37910d3e7a38c6e2bb9052058a511ab0d9c57241adff48d66c8f91f54d: Status 404 returned error can't find the container with id dbc80c37910d3e7a38c6e2bb9052058a511ab0d9c57241adff48d66c8f91f54d Jul 03 10:37:18.974466 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.974382 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" event={"ID":"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807","Type":"ContainerStarted","Data":"d6f9924fa3d0c2a3b80043d1a20fbffb74b003c900cbac74b99adf60948c1e07"} Jul 03 10:37:18.974466 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.974427 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" event={"ID":"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807","Type":"ContainerStarted","Data":"dbc80c37910d3e7a38c6e2bb9052058a511ab0d9c57241adff48d66c8f91f54d"} Jul 03 10:37:18.993671 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:18.993625 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" podStartSLOduration=0.993606051 podStartE2EDuration="993.606051ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:37:18.992885172 +0000 UTC m=+1910.031031780" watchObservedRunningTime="2026-07-03 10:37:18.993606051 +0000 UTC m=+1910.031752659" Jul 03 10:37:22.538410 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:22.538366 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:37:26.766636 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:26.766598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:37:26.867262 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:26.867206 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:37:28.630119 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:28.630070 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:28.630624 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:28.630137 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:28.643357 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:28.643326 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:29.037315 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:29.037232 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:37:57.868120 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:37:57.868091 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:38:01.411060 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:01.411022 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:38:14.199023 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:14.198979 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:38:16.755869 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:16.755835 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg"] Jul 03 10:38:16.794375 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:16.794325 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:38:17.710450 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:17.710404 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:38:17.867555 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:17.867520 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:38:18.281233 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:18.281192 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:38:18.465828 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:18.465791 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:38:18.564356 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:18.564276 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:38:18.745563 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:18.745527 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:38:23.360319 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:23.360213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:38:29.851766 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:29.851732 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:38:31.776179 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:31.776135 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" podUID="9b4d9aa9-cfe0-4fd0-9908-cfeee0946807" containerName="main" containerID="cri-o://d6f9924fa3d0c2a3b80043d1a20fbffb74b003c900cbac74b99adf60948c1e07" gracePeriod=45 Jul 03 10:38:32.022744 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.022718 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:38:32.170032 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.169989 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-dshm\") pod \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " Jul 03 10:38:32.170246 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.170085 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-home\") pod \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " Jul 03 10:38:32.170246 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.170117 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-model-cache\") pod \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " Jul 03 10:38:32.170246 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.170161 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-tmp-dir\") pod \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " Jul 03 10:38:32.170246 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.170205 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jvwh\" (UniqueName: \"kubernetes.io/projected/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-kube-api-access-4jvwh\") pod \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " Jul 03 10:38:32.170481 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.170247 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-tls-certs\") pod \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\" (UID: \"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807\") " Jul 03 10:38:32.170481 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.170420 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b4d9aa9-cfe0-4fd0-9908-cfeee0946807" (UID: "9b4d9aa9-cfe0-4fd0-9908-cfeee0946807"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:38:32.170481 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.170453 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-home" (OuterVolumeSpecName: "home") pod "9b4d9aa9-cfe0-4fd0-9908-cfeee0946807" (UID: "9b4d9aa9-cfe0-4fd0-9908-cfeee0946807"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:38:32.170604 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.170512 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b4d9aa9-cfe0-4fd0-9908-cfeee0946807" (UID: "9b4d9aa9-cfe0-4fd0-9908-cfeee0946807"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:38:32.170604 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.170572 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-home\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:38:32.170604 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.170592 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-model-cache\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:38:32.172385 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.172360 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-dshm" (OuterVolumeSpecName: "dshm") pod "9b4d9aa9-cfe0-4fd0-9908-cfeee0946807" (UID: "9b4d9aa9-cfe0-4fd0-9908-cfeee0946807"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:38:32.172499 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.172408 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b4d9aa9-cfe0-4fd0-9908-cfeee0946807" (UID: "9b4d9aa9-cfe0-4fd0-9908-cfeee0946807"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 10:38:32.172499 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.172472 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-kube-api-access-4jvwh" (OuterVolumeSpecName: "kube-api-access-4jvwh") pod "9b4d9aa9-cfe0-4fd0-9908-cfeee0946807" (UID: "9b4d9aa9-cfe0-4fd0-9908-cfeee0946807"). InnerVolumeSpecName "kube-api-access-4jvwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 10:38:32.271015 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.270977 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-tmp-dir\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:38:32.271015 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.271007 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jvwh\" (UniqueName: \"kubernetes.io/projected/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-kube-api-access-4jvwh\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:38:32.271015 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.271020 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-tls-certs\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:38:32.271280 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.271030 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807-dshm\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:38:32.319792 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.319755 2579 generic.go:356] "Generic (PLEG): container finished" podID="9b4d9aa9-cfe0-4fd0-9908-cfeee0946807" containerID="d6f9924fa3d0c2a3b80043d1a20fbffb74b003c900cbac74b99adf60948c1e07" exitCode=0 Jul 03 10:38:32.319994 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.319797 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" Jul 03 10:38:32.319994 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.319834 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" event={"ID":"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807","Type":"ContainerDied","Data":"d6f9924fa3d0c2a3b80043d1a20fbffb74b003c900cbac74b99adf60948c1e07"} Jul 03 10:38:32.319994 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.319881 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg" event={"ID":"9b4d9aa9-cfe0-4fd0-9908-cfeee0946807","Type":"ContainerDied","Data":"dbc80c37910d3e7a38c6e2bb9052058a511ab0d9c57241adff48d66c8f91f54d"} Jul 03 10:38:32.319994 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.319901 2579 scope.go:117] "RemoveContainer" containerID="d6f9924fa3d0c2a3b80043d1a20fbffb74b003c900cbac74b99adf60948c1e07" Jul 03 10:38:32.329046 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.329030 2579 scope.go:117] "RemoveContainer" containerID="d6f9924fa3d0c2a3b80043d1a20fbffb74b003c900cbac74b99adf60948c1e07" Jul 03 10:38:32.329351 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:38:32.329333 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6f9924fa3d0c2a3b80043d1a20fbffb74b003c900cbac74b99adf60948c1e07\": container with ID starting with d6f9924fa3d0c2a3b80043d1a20fbffb74b003c900cbac74b99adf60948c1e07 not found: ID does not exist" containerID="d6f9924fa3d0c2a3b80043d1a20fbffb74b003c900cbac74b99adf60948c1e07" Jul 03 10:38:32.329407 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.329359 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f9924fa3d0c2a3b80043d1a20fbffb74b003c900cbac74b99adf60948c1e07"} err="failed to get container status \"d6f9924fa3d0c2a3b80043d1a20fbffb74b003c900cbac74b99adf60948c1e07\": rpc error: code = NotFound desc = could not find container \"d6f9924fa3d0c2a3b80043d1a20fbffb74b003c900cbac74b99adf60948c1e07\": container with ID starting with d6f9924fa3d0c2a3b80043d1a20fbffb74b003c900cbac74b99adf60948c1e07 not found: ID does not exist" Jul 03 10:38:32.341128 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.341105 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg"] Jul 03 10:38:32.345790 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:32.345769 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1d628f/test-model-1d628f-kserve-5d7449cfd7-svllg"] Jul 03 10:38:33.530722 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:33.530686 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b4d9aa9-cfe0-4fd0-9908-cfeee0946807" path="/var/lib/kubelet/pods/9b4d9aa9-cfe0-4fd0-9908-cfeee0946807/volumes" Jul 03 10:38:55.243010 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:38:55.242965 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:01.460126 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:01.460088 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:01.762172 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:01.762084 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:01.987850 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:01.987821 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:02.326686 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:02.326653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:02.786542 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:02.786509 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:02.977840 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:02.977808 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:03.074514 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:03.074424 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:03.267494 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:03.267466 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:05.714688 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:05.714648 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:39:09.114137 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:09.114106 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:09.253891 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:09.253854 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:09.727644 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:09.727613 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:09.828821 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:09.828785 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:10.336199 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:10.336166 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:10.446086 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:10.446048 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:10.541613 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:10.541580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:14.143899 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:14.143838 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:14.676589 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:14.676554 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:14.878446 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:14.878412 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:15.024559 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:15.024475 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:15.128014 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:15.127981 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:18.551618 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:18.551583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:18.892881 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:18.892849 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:39:19.015953 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:19.015914 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:19.335706 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:19.335615 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:19.933733 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:19.933694 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:22.086485 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:22.086450 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:22.186940 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:22.186910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:22.307856 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:22.307814 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:22.645900 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:22.645864 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:23.039500 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:23.039424 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:23.162487 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:23.162452 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:26.540298 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:26.540260 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:29.692098 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:29.692058 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:39:34.737764 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:34.737727 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:35.199657 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:35.199610 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:35.360549 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:35.360512 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:35.460502 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:35.460415 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:35.635933 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:35.635884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:40.898246 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:40.898196 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:39:41.007628 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:41.007594 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:41.077187 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:41.077149 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:41.273990 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:41.273911 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:41.476079 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:41.476045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:41.734969 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:41.734931 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:42.195328 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:42.195296 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:42.361385 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:42.361345 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:42.463299 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:42.463180 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:42.746575 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:42.746496 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:48.332088 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:48.332050 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:48.433039 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:48.432993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:48.551346 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:48.551297 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:48.756082 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:48.756052 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:49.076859 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:49.076770 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:49.400892 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:49.400829 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:49.520473 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:49.520439 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:50.706829 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:50.706787 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:50.796298 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:50.796259 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:39:50.799590 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:50.799555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:52.166245 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:52.166200 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:52.253956 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:52.253924 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:52.324609 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:52.324577 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:52.851900 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:52.851867 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:52.955504 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:52.955470 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:56.664291 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:56.664258 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:56.981450 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:56.981370 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:57.292964 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:57.292860 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:39:58.081868 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:39:58.081833 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:40:00.189774 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:00.189740 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:40:00.283759 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:00.283724 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:40:00.373229 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:00.373171 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:40:00.492354 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:00.492262 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:40:00.996800 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:00.996763 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:40:01.053432 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:01.053394 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:40:01.153456 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:01.153424 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:40:04.671028 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:04.670991 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:40:09.502307 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:09.502271 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:40:13.761785 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:13.761736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:40:13.860789 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:13.860752 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:40:18.440835 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:18.440799 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:40:21.929234 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:21.929186 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:40:21.967750 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:21.967718 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:40:22.072769 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:22.072735 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:40:32.264359 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:32.264324 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:40:37.629510 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:37.629474 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:40:39.150424 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:39.150391 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:40:50.358948 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:50.358914 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:40:57.409484 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.409453 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g"] Jul 03 10:40:57.409963 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.409943 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b4d9aa9-cfe0-4fd0-9908-cfeee0946807" containerName="main" Jul 03 10:40:57.410043 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.409968 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b4d9aa9-cfe0-4fd0-9908-cfeee0946807" containerName="main" Jul 03 10:40:57.410097 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.410089 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b4d9aa9-cfe0-4fd0-9908-cfeee0946807" containerName="main" Jul 03 10:40:57.413003 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.412983 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.415472 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.415448 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-62c45e\"/\"kube-root-ca.crt\"" Jul 03 10:40:57.415472 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.415462 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-62c45e\"/\"auth-test-model-62c45e-kserve-self-signed-certs\"" Jul 03 10:40:57.415653 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.415451 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-62c45e\"/\"default-dockercfg-9f2pm\"" Jul 03 10:40:57.415653 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.415546 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-62c45e\"/\"openshift-service-ca.crt\"" Jul 03 10:40:57.422085 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.422062 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g"] Jul 03 10:40:57.439393 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.439367 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:40:57.500967 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.500931 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-dshm\") pod \"auth-test-model-62c45e-kserve-677c4b4c-2n94g\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.501164 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.500979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-model-cache\") pod \"auth-test-model-62c45e-kserve-677c4b4c-2n94g\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.501164 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.501006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-tmp-dir\") pod \"auth-test-model-62c45e-kserve-677c4b4c-2n94g\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.501164 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.501036 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73782414-e8dd-41ee-a744-869e9d96f3a1-tls-certs\") pod \"auth-test-model-62c45e-kserve-677c4b4c-2n94g\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.501164 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.501061 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pr7s\" (UniqueName: \"kubernetes.io/projected/73782414-e8dd-41ee-a744-869e9d96f3a1-kube-api-access-5pr7s\") pod \"auth-test-model-62c45e-kserve-677c4b4c-2n94g\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.501164 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.501113 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-home\") pod \"auth-test-model-62c45e-kserve-677c4b4c-2n94g\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.602355 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.602320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-dshm\") pod \"auth-test-model-62c45e-kserve-677c4b4c-2n94g\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.602539 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.602372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-model-cache\") pod \"auth-test-model-62c45e-kserve-677c4b4c-2n94g\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.602539 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.602389 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-tmp-dir\") pod \"auth-test-model-62c45e-kserve-677c4b4c-2n94g\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.602539 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.602408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73782414-e8dd-41ee-a744-869e9d96f3a1-tls-certs\") pod \"auth-test-model-62c45e-kserve-677c4b4c-2n94g\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.602670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.602577 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pr7s\" (UniqueName: \"kubernetes.io/projected/73782414-e8dd-41ee-a744-869e9d96f3a1-kube-api-access-5pr7s\") pod \"auth-test-model-62c45e-kserve-677c4b4c-2n94g\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.602670 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.602619 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-home\") pod \"auth-test-model-62c45e-kserve-677c4b4c-2n94g\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.602839 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.602806 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-tmp-dir\") pod \"auth-test-model-62c45e-kserve-677c4b4c-2n94g\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.602899 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.602841 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-model-cache\") pod \"auth-test-model-62c45e-kserve-677c4b4c-2n94g\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.602980 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.602963 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-home\") pod \"auth-test-model-62c45e-kserve-677c4b4c-2n94g\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.604719 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.604696 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-dshm\") pod \"auth-test-model-62c45e-kserve-677c4b4c-2n94g\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.604980 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.604963 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73782414-e8dd-41ee-a744-869e9d96f3a1-tls-certs\") pod \"auth-test-model-62c45e-kserve-677c4b4c-2n94g\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.611439 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.611416 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pr7s\" (UniqueName: \"kubernetes.io/projected/73782414-e8dd-41ee-a744-869e9d96f3a1-kube-api-access-5pr7s\") pod \"auth-test-model-62c45e-kserve-677c4b4c-2n94g\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.723482 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.723383 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:40:57.857033 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.857008 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g"] Jul 03 10:40:57.859302 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:40:57.859271 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73782414_e8dd_41ee_a744_869e9d96f3a1.slice/crio-c6a8d580a2c752d32be166fbfc422849f7e66480149d9e97c7fab3f57766da7c WatchSource:0}: Error finding container c6a8d580a2c752d32be166fbfc422849f7e66480149d9e97c7fab3f57766da7c: Status 404 returned error can't find the container with id c6a8d580a2c752d32be166fbfc422849f7e66480149d9e97c7fab3f57766da7c Jul 03 10:40:57.997275 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.997173 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" event={"ID":"73782414-e8dd-41ee-a744-869e9d96f3a1","Type":"ContainerStarted","Data":"ec1621b5564f86326fcc53b08c42e31516b090de2d16379efe2ded0fba5afa79"} Jul 03 10:40:57.997275 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:57.997231 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" event={"ID":"73782414-e8dd-41ee-a744-869e9d96f3a1","Type":"ContainerStarted","Data":"c6a8d580a2c752d32be166fbfc422849f7e66480149d9e97c7fab3f57766da7c"} Jul 03 10:40:58.018512 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:40:58.018460 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" podStartSLOduration=1.018443029 podStartE2EDuration="1.018443029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:40:58.0166625 +0000 UTC m=+2129.054809118" watchObservedRunningTime="2026-07-03 10:40:58.018443029 +0000 UTC m=+2129.056589636" Jul 03 10:41:01.033241 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:41:01.033193 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:41:07.724108 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:41:07.724064 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:41:07.724651 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:41:07.724121 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:41:07.737303 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:41:07.737277 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:41:08.055241 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:41:08.055150 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:41:39.429813 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:41:39.429776 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:41:49.356547 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:41:49.356502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:41:49.425589 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:41:49.425556 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:41:54.601294 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:41:54.601252 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:42:07.802357 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:42:07.802321 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:42:24.597235 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:42:24.597196 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:42:41.799525 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:42:41.799487 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:42:58.199571 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:42:58.199491 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:43:14.793981 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:14.793938 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:43:19.596398 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:19.596364 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:43:20.478039 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:20.478004 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g"] Jul 03 10:43:20.506235 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:20.506198 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:43:28.722707 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:28.722664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:43:33.582179 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:33.582128 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:43:35.498029 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.497966 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" podUID="73782414-e8dd-41ee-a744-869e9d96f3a1" containerName="main" containerID="cri-o://ec1621b5564f86326fcc53b08c42e31516b090de2d16379efe2ded0fba5afa79" gracePeriod=45 Jul 03 10:43:35.733821 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.733786 2579 generic.go:356] "Generic (PLEG): container finished" podID="73782414-e8dd-41ee-a744-869e9d96f3a1" containerID="ec1621b5564f86326fcc53b08c42e31516b090de2d16379efe2ded0fba5afa79" exitCode=0 Jul 03 10:43:35.733992 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.733895 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" event={"ID":"73782414-e8dd-41ee-a744-869e9d96f3a1","Type":"ContainerDied","Data":"ec1621b5564f86326fcc53b08c42e31516b090de2d16379efe2ded0fba5afa79"} Jul 03 10:43:35.755441 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.755388 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:43:35.773717 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.773689 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-tmp-dir\") pod \"73782414-e8dd-41ee-a744-869e9d96f3a1\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " Jul 03 10:43:35.773835 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.773728 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-dshm\") pod \"73782414-e8dd-41ee-a744-869e9d96f3a1\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " Jul 03 10:43:35.773835 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.773752 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-home\") pod \"73782414-e8dd-41ee-a744-869e9d96f3a1\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " Jul 03 10:43:35.773835 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.773800 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pr7s\" (UniqueName: \"kubernetes.io/projected/73782414-e8dd-41ee-a744-869e9d96f3a1-kube-api-access-5pr7s\") pod \"73782414-e8dd-41ee-a744-869e9d96f3a1\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " Jul 03 10:43:35.773835 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.773818 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-model-cache\") pod \"73782414-e8dd-41ee-a744-869e9d96f3a1\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " Jul 03 10:43:35.774054 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.773853 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73782414-e8dd-41ee-a744-869e9d96f3a1-tls-certs\") pod \"73782414-e8dd-41ee-a744-869e9d96f3a1\" (UID: \"73782414-e8dd-41ee-a744-869e9d96f3a1\") " Jul 03 10:43:35.774054 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.773938 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "73782414-e8dd-41ee-a744-869e9d96f3a1" (UID: "73782414-e8dd-41ee-a744-869e9d96f3a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:43:35.774054 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.774017 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-home" (OuterVolumeSpecName: "home") pod "73782414-e8dd-41ee-a744-869e9d96f3a1" (UID: "73782414-e8dd-41ee-a744-869e9d96f3a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:43:35.774204 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.774105 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "73782414-e8dd-41ee-a744-869e9d96f3a1" (UID: "73782414-e8dd-41ee-a744-869e9d96f3a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:43:35.774204 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.774164 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-tmp-dir\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:43:35.774204 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.774177 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-home\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:43:35.774204 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.774185 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-model-cache\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:43:35.775947 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.775924 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73782414-e8dd-41ee-a744-869e9d96f3a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "73782414-e8dd-41ee-a744-869e9d96f3a1" (UID: "73782414-e8dd-41ee-a744-869e9d96f3a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 10:43:35.775947 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.775937 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-dshm" (OuterVolumeSpecName: "dshm") pod "73782414-e8dd-41ee-a744-869e9d96f3a1" (UID: "73782414-e8dd-41ee-a744-869e9d96f3a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:43:35.776090 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.776014 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73782414-e8dd-41ee-a744-869e9d96f3a1-kube-api-access-5pr7s" (OuterVolumeSpecName: "kube-api-access-5pr7s") pod "73782414-e8dd-41ee-a744-869e9d96f3a1" (UID: "73782414-e8dd-41ee-a744-869e9d96f3a1"). InnerVolumeSpecName "kube-api-access-5pr7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 10:43:35.874900 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.874866 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73782414-e8dd-41ee-a744-869e9d96f3a1-dshm\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:43:35.874900 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.874897 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pr7s\" (UniqueName: \"kubernetes.io/projected/73782414-e8dd-41ee-a744-869e9d96f3a1-kube-api-access-5pr7s\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:43:35.874900 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:35.874908 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73782414-e8dd-41ee-a744-869e9d96f3a1-tls-certs\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:43:36.739233 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:36.739195 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" Jul 03 10:43:36.739708 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:36.739241 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g" event={"ID":"73782414-e8dd-41ee-a744-869e9d96f3a1","Type":"ContainerDied","Data":"c6a8d580a2c752d32be166fbfc422849f7e66480149d9e97c7fab3f57766da7c"} Jul 03 10:43:36.739708 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:36.739285 2579 scope.go:117] "RemoveContainer" containerID="ec1621b5564f86326fcc53b08c42e31516b090de2d16379efe2ded0fba5afa79" Jul 03 10:43:36.771038 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:36.771013 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g"] Jul 03 10:43:36.779655 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:36.779632 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-62c45e/auth-test-model-62c45e-kserve-677c4b4c-2n94g"] Jul 03 10:43:37.533068 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:37.533026 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73782414-e8dd-41ee-a744-869e9d96f3a1" path="/var/lib/kubelet/pods/73782414-e8dd-41ee-a744-869e9d96f3a1/volumes" Jul 03 10:43:55.300876 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.300841 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564"] Jul 03 10:43:55.301384 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.301365 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73782414-e8dd-41ee-a744-869e9d96f3a1" containerName="main" Jul 03 10:43:55.301461 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.301387 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="73782414-e8dd-41ee-a744-869e9d96f3a1" containerName="main" Jul 03 10:43:55.301512 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.301472 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="73782414-e8dd-41ee-a744-869e9d96f3a1" containerName="main" Jul 03 10:43:55.305754 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.305734 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.309475 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.309449 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-62c45e\"/\"kube-root-ca.crt\"" Jul 03 10:43:55.311109 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.311084 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-62c45e\"/\"sub-test-model-62c45e-kserve-self-signed-certs\"" Jul 03 10:43:55.311235 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.311128 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-62c45e\"/\"openshift-service-ca.crt\"" Jul 03 10:43:55.311235 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.311140 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-62c45e\"/\"default-dockercfg-9f2pm\"" Jul 03 10:43:55.323374 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.323349 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564"] Jul 03 10:43:55.378654 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.378620 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:43:55.441529 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.441497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-dshm\") pod \"sub-test-model-62c45e-kserve-69b4b6597-9j564\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.441710 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.441560 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgjkl\" (UniqueName: \"kubernetes.io/projected/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-kube-api-access-jgjkl\") pod \"sub-test-model-62c45e-kserve-69b4b6597-9j564\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.441710 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.441594 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-model-cache\") pod \"sub-test-model-62c45e-kserve-69b4b6597-9j564\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.441710 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.441619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-tls-certs\") pod \"sub-test-model-62c45e-kserve-69b4b6597-9j564\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.441710 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.441647 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-home\") pod \"sub-test-model-62c45e-kserve-69b4b6597-9j564\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.441857 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.441723 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-tmp-dir\") pod \"sub-test-model-62c45e-kserve-69b4b6597-9j564\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.542695 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.542662 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-tmp-dir\") pod \"sub-test-model-62c45e-kserve-69b4b6597-9j564\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.542695 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.542698 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-dshm\") pod \"sub-test-model-62c45e-kserve-69b4b6597-9j564\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.542970 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.542738 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgjkl\" (UniqueName: \"kubernetes.io/projected/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-kube-api-access-jgjkl\") pod \"sub-test-model-62c45e-kserve-69b4b6597-9j564\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.542970 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.542762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-model-cache\") pod \"sub-test-model-62c45e-kserve-69b4b6597-9j564\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.542970 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.542782 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-tls-certs\") pod \"sub-test-model-62c45e-kserve-69b4b6597-9j564\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.542970 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.542815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-home\") pod \"sub-test-model-62c45e-kserve-69b4b6597-9j564\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.543182 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.543128 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-tmp-dir\") pod \"sub-test-model-62c45e-kserve-69b4b6597-9j564\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.543182 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.543166 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-model-cache\") pod \"sub-test-model-62c45e-kserve-69b4b6597-9j564\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.543286 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.543236 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-home\") pod \"sub-test-model-62c45e-kserve-69b4b6597-9j564\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.545089 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.545072 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-dshm\") pod \"sub-test-model-62c45e-kserve-69b4b6597-9j564\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.545386 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.545360 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-tls-certs\") pod \"sub-test-model-62c45e-kserve-69b4b6597-9j564\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.553145 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.553072 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgjkl\" (UniqueName: \"kubernetes.io/projected/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-kube-api-access-jgjkl\") pod \"sub-test-model-62c45e-kserve-69b4b6597-9j564\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.615954 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.615922 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:43:55.768124 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.768094 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564"] Jul 03 10:43:55.769549 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:43:55.769519 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bdfe7ae_3697_42b2_a57f_add9bcb25a8f.slice/crio-083ce67a0109dd20b9a02a976b3b91c3a1a94a03bd1379e5f6e173e5a3aaa5e6 WatchSource:0}: Error finding container 083ce67a0109dd20b9a02a976b3b91c3a1a94a03bd1379e5f6e173e5a3aaa5e6: Status 404 returned error can't find the container with id 083ce67a0109dd20b9a02a976b3b91c3a1a94a03bd1379e5f6e173e5a3aaa5e6 Jul 03 10:43:55.826089 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:55.826063 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" event={"ID":"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f","Type":"ContainerStarted","Data":"083ce67a0109dd20b9a02a976b3b91c3a1a94a03bd1379e5f6e173e5a3aaa5e6"} Jul 03 10:43:56.833288 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:56.833245 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" event={"ID":"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f","Type":"ContainerStarted","Data":"62fd4486bba9c9e6218820642ea14f576f8f8c3c8135b21de30a0786b7d6938d"} Jul 03 10:43:56.863848 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:56.863795 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" podStartSLOduration=1.8637785999999998 podStartE2EDuration="1.8637786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:43:56.863411054 +0000 UTC m=+2307.901557662" watchObservedRunningTime="2026-07-03 10:43:56.8637786 +0000 UTC m=+2307.901925206" Jul 03 10:43:59.815939 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:43:59.815900 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:44:05.616091 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:05.616041 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:44:05.616578 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:05.616103 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:44:05.628162 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:05.628135 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:44:05.893282 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:05.893192 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:44:18.402696 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:18.402657 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:44:33.322185 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:33.322148 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:44:38.618631 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:38.618595 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:44:39.422265 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:39.422204 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564"] Jul 03 10:44:39.444746 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:39.444717 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:44:50.741614 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:50.741575 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:44:51.561519 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:51.561479 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:44:51.663053 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:51.663016 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:44:54.176700 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.176658 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:44:54.444294 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.444190 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" podUID="1bdfe7ae-3697-42b2-a57f-add9bcb25a8f" containerName="main" containerID="cri-o://62fd4486bba9c9e6218820642ea14f576f8f8c3c8135b21de30a0786b7d6938d" gracePeriod=45 Jul 03 10:44:54.696918 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.696857 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:44:54.768166 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.768136 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgjkl\" (UniqueName: \"kubernetes.io/projected/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-kube-api-access-jgjkl\") pod \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " Jul 03 10:44:54.768365 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.768180 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-tmp-dir\") pod \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " Jul 03 10:44:54.768365 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.768249 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-model-cache\") pod \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " Jul 03 10:44:54.768365 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.768305 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-home\") pod \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " Jul 03 10:44:54.768365 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.768342 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-dshm\") pod \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " Jul 03 10:44:54.768590 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.768384 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-tls-certs\") pod \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\" (UID: \"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f\") " Jul 03 10:44:54.768590 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.768521 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1bdfe7ae-3697-42b2-a57f-add9bcb25a8f" (UID: "1bdfe7ae-3697-42b2-a57f-add9bcb25a8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:44:54.768676 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.768578 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-model-cache" (OuterVolumeSpecName: "model-cache") pod "1bdfe7ae-3697-42b2-a57f-add9bcb25a8f" (UID: "1bdfe7ae-3697-42b2-a57f-add9bcb25a8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:44:54.768676 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.768592 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-home" (OuterVolumeSpecName: "home") pod "1bdfe7ae-3697-42b2-a57f-add9bcb25a8f" (UID: "1bdfe7ae-3697-42b2-a57f-add9bcb25a8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:44:54.768676 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.768638 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-tmp-dir\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:44:54.770379 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.770353 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-dshm" (OuterVolumeSpecName: "dshm") pod "1bdfe7ae-3697-42b2-a57f-add9bcb25a8f" (UID: "1bdfe7ae-3697-42b2-a57f-add9bcb25a8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:44:54.770462 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.770359 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-kube-api-access-jgjkl" (OuterVolumeSpecName: "kube-api-access-jgjkl") pod "1bdfe7ae-3697-42b2-a57f-add9bcb25a8f" (UID: "1bdfe7ae-3697-42b2-a57f-add9bcb25a8f"). InnerVolumeSpecName "kube-api-access-jgjkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 10:44:54.770503 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.770474 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1bdfe7ae-3697-42b2-a57f-add9bcb25a8f" (UID: "1bdfe7ae-3697-42b2-a57f-add9bcb25a8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 10:44:54.869072 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.869037 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-model-cache\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:44:54.869072 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.869064 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-home\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:44:54.869072 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.869073 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-dshm\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:44:54.869072 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.869083 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-tls-certs\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:44:54.869371 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:54.869091 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgjkl\" (UniqueName: \"kubernetes.io/projected/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f-kube-api-access-jgjkl\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:44:55.107604 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:55.107568 2579 generic.go:356] "Generic (PLEG): container finished" podID="1bdfe7ae-3697-42b2-a57f-add9bcb25a8f" containerID="62fd4486bba9c9e6218820642ea14f576f8f8c3c8135b21de30a0786b7d6938d" exitCode=0 Jul 03 10:44:55.107783 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:55.107612 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" event={"ID":"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f","Type":"ContainerDied","Data":"62fd4486bba9c9e6218820642ea14f576f8f8c3c8135b21de30a0786b7d6938d"} Jul 03 10:44:55.107783 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:55.107642 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" event={"ID":"1bdfe7ae-3697-42b2-a57f-add9bcb25a8f","Type":"ContainerDied","Data":"083ce67a0109dd20b9a02a976b3b91c3a1a94a03bd1379e5f6e173e5a3aaa5e6"} Jul 03 10:44:55.107783 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:55.107659 2579 scope.go:117] "RemoveContainer" containerID="62fd4486bba9c9e6218820642ea14f576f8f8c3c8135b21de30a0786b7d6938d" Jul 03 10:44:55.107783 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:55.107615 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564" Jul 03 10:44:55.116503 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:55.116484 2579 scope.go:117] "RemoveContainer" containerID="62fd4486bba9c9e6218820642ea14f576f8f8c3c8135b21de30a0786b7d6938d" Jul 03 10:44:55.116772 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:44:55.116748 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62fd4486bba9c9e6218820642ea14f576f8f8c3c8135b21de30a0786b7d6938d\": container with ID starting with 62fd4486bba9c9e6218820642ea14f576f8f8c3c8135b21de30a0786b7d6938d not found: ID does not exist" containerID="62fd4486bba9c9e6218820642ea14f576f8f8c3c8135b21de30a0786b7d6938d" Jul 03 10:44:55.116873 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:55.116779 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62fd4486bba9c9e6218820642ea14f576f8f8c3c8135b21de30a0786b7d6938d"} err="failed to get container status \"62fd4486bba9c9e6218820642ea14f576f8f8c3c8135b21de30a0786b7d6938d\": rpc error: code = NotFound desc = could not find container \"62fd4486bba9c9e6218820642ea14f576f8f8c3c8135b21de30a0786b7d6938d\": container with ID starting with 62fd4486bba9c9e6218820642ea14f576f8f8c3c8135b21de30a0786b7d6938d not found: ID does not exist" Jul 03 10:44:55.128170 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:55.128147 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564"] Jul 03 10:44:55.134599 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:55.134577 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-62c45e/sub-test-model-62c45e-kserve-69b4b6597-9j564"] Jul 03 10:44:55.531075 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:44:55.531002 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bdfe7ae-3697-42b2-a57f-add9bcb25a8f" path="/var/lib/kubelet/pods/1bdfe7ae-3697-42b2-a57f-add9bcb25a8f/volumes" Jul 03 10:45:05.401852 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:05.401810 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:45:14.735606 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:14.735568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:45:14.836635 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:14.836593 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:45:15.203503 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.203474 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v"] Jul 03 10:45:15.203804 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.203792 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bdfe7ae-3697-42b2-a57f-add9bcb25a8f" containerName="main" Jul 03 10:45:15.203849 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.203806 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bdfe7ae-3697-42b2-a57f-add9bcb25a8f" containerName="main" Jul 03 10:45:15.203910 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.203901 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bdfe7ae-3697-42b2-a57f-add9bcb25a8f" containerName="main" Jul 03 10:45:15.206976 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.206960 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.209185 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.209158 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-62c45e\"/\"openshift-service-ca.crt\"" Jul 03 10:45:15.209702 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.209682 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-62c45e\"/\"default-dockercfg-9f2pm\"" Jul 03 10:45:15.209764 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.209732 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-62c45e\"/\"kube-root-ca.crt\"" Jul 03 10:45:15.209817 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.209798 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-62c45e\"/\"rate-test-model-62c45e-kserve-self-signed-certs\"" Jul 03 10:45:15.216201 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.216179 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v"] Jul 03 10:45:15.223976 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.223951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:45:15.240817 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.240790 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-home\") pod \"rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.240965 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.240822 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m6ff\" (UniqueName: \"kubernetes.io/projected/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-kube-api-access-2m6ff\") pod \"rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.240965 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.240847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-tls-certs\") pod \"rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.240965 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.240931 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-dshm\") pod \"rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.241100 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.240968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-model-cache\") pod \"rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.241100 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.240992 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-tmp-dir\") pod \"rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.341398 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.341360 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-dshm\") pod \"rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.341595 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.341414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-model-cache\") pod \"rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.341595 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.341446 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-tmp-dir\") pod \"rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.341595 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.341498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-home\") pod \"rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.341595 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.341522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m6ff\" (UniqueName: \"kubernetes.io/projected/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-kube-api-access-2m6ff\") pod \"rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.341595 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.341552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-tls-certs\") pod \"rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.341931 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.341893 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-model-cache\") pod \"rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.341931 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.341927 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-tmp-dir\") pod \"rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.342090 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.341987 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-home\") pod \"rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.343641 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.343619 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-dshm\") pod \"rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.344029 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.344012 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-tls-certs\") pod \"rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.349143 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.349119 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m6ff\" (UniqueName: \"kubernetes.io/projected/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-kube-api-access-2m6ff\") pod \"rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.517442 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.517357 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:15.663950 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:15.663915 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v"] Jul 03 10:45:15.666884 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:45:15.666851 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a538522_03dd_4c2a_a0e7_cd60a95e1b59.slice/crio-3f09ce9c330f57158ec01968b5faadbecee0163fa3d813b81a304e10fcee6691 WatchSource:0}: Error finding container 3f09ce9c330f57158ec01968b5faadbecee0163fa3d813b81a304e10fcee6691: Status 404 returned error can't find the container with id 3f09ce9c330f57158ec01968b5faadbecee0163fa3d813b81a304e10fcee6691 Jul 03 10:45:16.207974 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:16.207936 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" event={"ID":"3a538522-03dd-4c2a-a0e7-cd60a95e1b59","Type":"ContainerStarted","Data":"be0295f6a05ae45cf0b3577417cd5a7c5fcf73fdfb69eab24675d6a920c089ee"} Jul 03 10:45:16.207974 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:16.207978 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" event={"ID":"3a538522-03dd-4c2a-a0e7-cd60a95e1b59","Type":"ContainerStarted","Data":"3f09ce9c330f57158ec01968b5faadbecee0163fa3d813b81a304e10fcee6691"} Jul 03 10:45:16.228004 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:16.227954 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" podStartSLOduration=1.227938141 podStartE2EDuration="1.227938141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 10:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:45:16.226851094 +0000 UTC m=+2387.264997727" watchObservedRunningTime="2026-07-03 10:45:16.227938141 +0000 UTC m=+2387.266084769" Jul 03 10:45:20.999924 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:20.999886 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:45:22.730133 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:22.730099 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:45:22.829680 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:22.829643 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:45:25.517953 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:25.517915 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:25.517953 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:25.517965 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:25.532599 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:25.532571 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:26.267742 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:26.267714 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:45:34.920800 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:34.920765 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:45:47.324523 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:47.324491 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:45:49.392070 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:49.392039 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:45:55.586942 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:55.586832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:45:56.574384 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:56.574353 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v"] Jul 03 10:45:57.273664 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:57.273629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:45:57.618624 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:57.618595 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:45:58.003258 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:58.003156 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:45:58.104076 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:58.104044 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:45:58.243313 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:45:58.243284 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:46:01.990648 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:01.990615 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:46:03.137618 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:03.137580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:46:11.594262 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:11.594183 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" podUID="3a538522-03dd-4c2a-a0e7-cd60a95e1b59" containerName="main" containerID="cri-o://be0295f6a05ae45cf0b3577417cd5a7c5fcf73fdfb69eab24675d6a920c089ee" gracePeriod=45 Jul 03 10:46:11.849027 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:11.849005 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:46:11.939099 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:11.939064 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-home\") pod \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " Jul 03 10:46:11.939099 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:11.939105 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m6ff\" (UniqueName: \"kubernetes.io/projected/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-kube-api-access-2m6ff\") pod \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " Jul 03 10:46:11.939410 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:11.939128 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-dshm\") pod \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " Jul 03 10:46:11.939410 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:11.939147 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-tmp-dir\") pod \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " Jul 03 10:46:11.939410 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:11.939261 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-tls-certs\") pod \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " Jul 03 10:46:11.939410 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:11.939337 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-model-cache\") pod \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\" (UID: \"3a538522-03dd-4c2a-a0e7-cd60a95e1b59\") " Jul 03 10:46:11.939629 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:11.939455 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a538522-03dd-4c2a-a0e7-cd60a95e1b59" (UID: "3a538522-03dd-4c2a-a0e7-cd60a95e1b59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:46:11.939629 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:11.939473 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-home" (OuterVolumeSpecName: "home") pod "3a538522-03dd-4c2a-a0e7-cd60a95e1b59" (UID: "3a538522-03dd-4c2a-a0e7-cd60a95e1b59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:46:11.939629 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:11.939560 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a538522-03dd-4c2a-a0e7-cd60a95e1b59" (UID: "3a538522-03dd-4c2a-a0e7-cd60a95e1b59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:46:11.939795 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:11.939666 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-home\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:46:11.939795 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:11.939685 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-tmp-dir\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:46:11.939795 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:11.939699 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-model-cache\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:46:11.941386 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:11.941361 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-dshm" (OuterVolumeSpecName: "dshm") pod "3a538522-03dd-4c2a-a0e7-cd60a95e1b59" (UID: "3a538522-03dd-4c2a-a0e7-cd60a95e1b59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 10:46:11.941386 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:11.941363 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-kube-api-access-2m6ff" (OuterVolumeSpecName: "kube-api-access-2m6ff") pod "3a538522-03dd-4c2a-a0e7-cd60a95e1b59" (UID: "3a538522-03dd-4c2a-a0e7-cd60a95e1b59"). InnerVolumeSpecName "kube-api-access-2m6ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 10:46:11.941522 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:11.941362 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a538522-03dd-4c2a-a0e7-cd60a95e1b59" (UID: "3a538522-03dd-4c2a-a0e7-cd60a95e1b59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 10:46:12.040648 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:12.040603 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m6ff\" (UniqueName: \"kubernetes.io/projected/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-kube-api-access-2m6ff\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:46:12.040648 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:12.040644 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-dshm\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:46:12.040648 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:12.040655 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a538522-03dd-4c2a-a0e7-cd60a95e1b59-tls-certs\") on node \"ip-10-0-143-105.ec2.internal\" DevicePath \"\"" Jul 03 10:46:12.465762 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:12.465729 2579 generic.go:356] "Generic (PLEG): container finished" podID="3a538522-03dd-4c2a-a0e7-cd60a95e1b59" containerID="be0295f6a05ae45cf0b3577417cd5a7c5fcf73fdfb69eab24675d6a920c089ee" exitCode=0 Jul 03 10:46:12.465925 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:12.465777 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" Jul 03 10:46:12.465925 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:12.465818 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" event={"ID":"3a538522-03dd-4c2a-a0e7-cd60a95e1b59","Type":"ContainerDied","Data":"be0295f6a05ae45cf0b3577417cd5a7c5fcf73fdfb69eab24675d6a920c089ee"} Jul 03 10:46:12.465925 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:12.465864 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v" event={"ID":"3a538522-03dd-4c2a-a0e7-cd60a95e1b59","Type":"ContainerDied","Data":"3f09ce9c330f57158ec01968b5faadbecee0163fa3d813b81a304e10fcee6691"} Jul 03 10:46:12.465925 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:12.465886 2579 scope.go:117] "RemoveContainer" containerID="be0295f6a05ae45cf0b3577417cd5a7c5fcf73fdfb69eab24675d6a920c089ee" Jul 03 10:46:12.476717 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:12.476671 2579 scope.go:117] "RemoveContainer" containerID="be0295f6a05ae45cf0b3577417cd5a7c5fcf73fdfb69eab24675d6a920c089ee" Jul 03 10:46:12.477064 ip-10-0-143-105 kubenswrapper[2579]: E0703 10:46:12.477042 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be0295f6a05ae45cf0b3577417cd5a7c5fcf73fdfb69eab24675d6a920c089ee\": container with ID starting with be0295f6a05ae45cf0b3577417cd5a7c5fcf73fdfb69eab24675d6a920c089ee not found: ID does not exist" containerID="be0295f6a05ae45cf0b3577417cd5a7c5fcf73fdfb69eab24675d6a920c089ee" Jul 03 10:46:12.477122 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:12.477077 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0295f6a05ae45cf0b3577417cd5a7c5fcf73fdfb69eab24675d6a920c089ee"} err="failed to get container status \"be0295f6a05ae45cf0b3577417cd5a7c5fcf73fdfb69eab24675d6a920c089ee\": rpc error: code = NotFound desc = could not find container \"be0295f6a05ae45cf0b3577417cd5a7c5fcf73fdfb69eab24675d6a920c089ee\": container with ID starting with be0295f6a05ae45cf0b3577417cd5a7c5fcf73fdfb69eab24675d6a920c089ee not found: ID does not exist" Jul 03 10:46:12.487732 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:12.487709 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v"] Jul 03 10:46:12.492348 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:12.492328 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-62c45e/rate-test-model-62c45e-kserve-b4c7d67ff-8hm6v"] Jul 03 10:46:13.530435 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:13.530397 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a538522-03dd-4c2a-a0e7-cd60a95e1b59" path="/var/lib/kubelet/pods/3a538522-03dd-4c2a-a0e7-cd60a95e1b59/volumes" Jul 03 10:46:40.264848 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:40.264811 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:46:40.494212 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:40.494184 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:46:40.738157 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:40.738127 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:46:41.048114 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:41.048012 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:46:41.399531 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:41.399499 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:46:41.725011 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:41.724900 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:46:41.826714 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:41.826676 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:46:41.991797 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:41.991716 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:46:44.292520 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:44.292482 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tsqcn"] Jul 03 10:46:47.655696 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:47.655665 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:46:47.755438 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:47.755400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:46:47.921099 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:47.920981 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:46:48.164838 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:48.164799 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:46:48.572464 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:48.572431 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bcfbc4d69-ldgkh_03edaf58-b0b1-4aec-b60a-1cf296dcc1cd/authorino/0.log" Jul 03 10:46:53.065241 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:53.065186 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-s498m_e3467416-d3c8-461e-a5a2-a913adb30885/manager/0.log" Jul 03 10:46:53.187338 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:53.187300 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hpnsx_ce204869-bbec-4287-9139-a6c80de9fa81/manager/0.log" Jul 03 10:46:53.814996 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:53.814964 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-w9t6l_2bee4600-2b04-4cbd-b3fe-518f0ee736bf/server/0.log" Jul 03 10:46:53.929207 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:53.929180 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-r9nqc_c3f3093f-95db-461e-99ae-f8d3153e85bb/manager/0.log" Jul 03 10:46:54.162100 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:54.162071 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789547c8f-5trlf_07f9e9d4-65d3-4b97-a4eb-a88c47a707f8/copy-manifests/0.log" Jul 03 10:46:54.175068 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:54.175041 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789547c8f-5trlf_07f9e9d4-65d3-4b97-a4eb-a88c47a707f8/manager/0.log" Jul 03 10:46:55.736276 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:55.736244 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bcfbc4d69-ldgkh_03edaf58-b0b1-4aec-b60a-1cf296dcc1cd/authorino/0.log" Jul 03 10:46:55.855717 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:55.855679 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-mtd68_dff227d7-468e-4b1e-aebf-0d0ed417464a/manager/0.log" Jul 03 10:46:56.081696 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:56.081622 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-lvdb6_9404fda8-21f8-4527-aadf-0bf0bcce7316/kuadrant-console-plugin/0.log" Jul 03 10:46:56.200977 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:56.200943 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-kw9d4_9a06a4c9-5506-4190-b7b2-af804d7cc261/registry-server/0.log" Jul 03 10:46:56.322119 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:56.322093 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-wzjdz_b3a6d5e0-cc4f-44e8-8b82-85b55c9c5301/manager/0.log" Jul 03 10:46:56.432484 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:56.432455 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-tsqcn_a1c36347-b59d-4225-a2a5-1772d3433cb9/limitador/0.log" Jul 03 10:46:56.548614 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:56.548579 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-vxnfn_7e88a64e-c97f-43eb-83c4-583b034f38a9/manager/0.log" Jul 03 10:46:57.249492 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:57.249459 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b56494769-gmjkj_170f0891-9d61-4a9e-9211-4fcae92ddc58/kube-auth-proxy/0.log" Jul 03 10:46:58.516652 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:58.516616 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-bpdpq_6c3f7179-b886-40fd-abd8-6dec502217e0/main/0.log" Jul 03 10:46:58.626063 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:46:58.626032 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-n92tv_5876fe42-f276-4325-91c6-f6a60ed620a9/main/0.log" Jul 03 10:47:02.302453 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:02.302403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:47:02.400511 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:02.400479 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" limit=1024 Jul 03 10:47:05.307073 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:05.307041 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pzm5l_a6273a00-72c1-456e-b47d-9a24d012f8c0/global-pull-secret-syncer/0.log" Jul 03 10:47:05.439725 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:05.439689 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qskxt_b6e7cb03-364e-4480-84c8-e163592cfa64/konnectivity-agent/0.log" Jul 03 10:47:05.598888 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:05.598859 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-105.ec2.internal_6b144ac3fd0711db4037382f38ebdd87/haproxy/0.log" Jul 03 10:47:10.156711 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:10.156677 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6bcfbc4d69-ldgkh_03edaf58-b0b1-4aec-b60a-1cf296dcc1cd/authorino/0.log" Jul 03 10:47:10.213835 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:10.213803 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-mtd68_dff227d7-468e-4b1e-aebf-0d0ed417464a/manager/0.log" Jul 03 10:47:10.299082 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:10.299054 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-lvdb6_9404fda8-21f8-4527-aadf-0bf0bcce7316/kuadrant-console-plugin/0.log" Jul 03 10:47:10.359143 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:10.359114 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-kw9d4_9a06a4c9-5506-4190-b7b2-af804d7cc261/registry-server/0.log" Jul 03 10:47:10.514782 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:10.514697 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-wzjdz_b3a6d5e0-cc4f-44e8-8b82-85b55c9c5301/manager/0.log" Jul 03 10:47:10.577826 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:10.577800 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-tsqcn_a1c36347-b59d-4225-a2a5-1772d3433cb9/limitador/0.log" Jul 03 10:47:10.660618 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:10.660584 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-vxnfn_7e88a64e-c97f-43eb-83c4-583b034f38a9/manager/0.log" Jul 03 10:47:13.020108 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:13.020077 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cxfgx_eaaba771-633f-4753-a355-8c7e314a2b1a/node-exporter/0.log" Jul 03 10:47:13.041822 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:13.041795 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cxfgx_eaaba771-633f-4753-a355-8c7e314a2b1a/kube-rbac-proxy/0.log" Jul 03 10:47:13.069436 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:13.069409 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cxfgx_eaaba771-633f-4753-a355-8c7e314a2b1a/init-textfile/0.log" Jul 03 10:47:14.969561 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:14.969532 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-9hhgb_0319f4ec-7dd2-4ca9-9f58-1ba5e6f33ca7/networking-console-plugin/0.log" Jul 03 10:47:15.373265 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.373231 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47"] Jul 03 10:47:15.373572 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.373560 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a538522-03dd-4c2a-a0e7-cd60a95e1b59" containerName="main" Jul 03 10:47:15.373619 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.373573 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a538522-03dd-4c2a-a0e7-cd60a95e1b59" containerName="main" Jul 03 10:47:15.373658 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.373629 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a538522-03dd-4c2a-a0e7-cd60a95e1b59" containerName="main" Jul 03 10:47:15.376632 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.376612 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" Jul 03 10:47:15.378767 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.378740 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dwvz5\"/\"kube-root-ca.crt\"" Jul 03 10:47:15.378895 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.378791 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dwvz5\"/\"default-dockercfg-zrqnd\"" Jul 03 10:47:15.379035 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.379020 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dwvz5\"/\"openshift-service-ca.crt\"" Jul 03 10:47:15.384231 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.384130 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47"] Jul 03 10:47:15.456265 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.456209 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/835afa01-3c31-468c-992a-375962c6c2d6-proc\") pod \"perf-node-gather-daemonset-t4d47\" (UID: \"835afa01-3c31-468c-992a-375962c6c2d6\") " pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" Jul 03 10:47:15.456265 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.456267 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/835afa01-3c31-468c-992a-375962c6c2d6-podres\") pod \"perf-node-gather-daemonset-t4d47\" (UID: \"835afa01-3c31-468c-992a-375962c6c2d6\") " pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" Jul 03 10:47:15.456474 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.456341 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kbhh\" (UniqueName: \"kubernetes.io/projected/835afa01-3c31-468c-992a-375962c6c2d6-kube-api-access-2kbhh\") pod \"perf-node-gather-daemonset-t4d47\" (UID: \"835afa01-3c31-468c-992a-375962c6c2d6\") " pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" Jul 03 10:47:15.456474 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.456373 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/835afa01-3c31-468c-992a-375962c6c2d6-lib-modules\") pod \"perf-node-gather-daemonset-t4d47\" (UID: \"835afa01-3c31-468c-992a-375962c6c2d6\") " pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" Jul 03 10:47:15.456474 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.456424 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/835afa01-3c31-468c-992a-375962c6c2d6-sys\") pod \"perf-node-gather-daemonset-t4d47\" (UID: \"835afa01-3c31-468c-992a-375962c6c2d6\") " pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" Jul 03 10:47:15.557628 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.557597 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/835afa01-3c31-468c-992a-375962c6c2d6-podres\") pod \"perf-node-gather-daemonset-t4d47\" (UID: \"835afa01-3c31-468c-992a-375962c6c2d6\") " pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" Jul 03 10:47:15.557797 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.557640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kbhh\" (UniqueName: \"kubernetes.io/projected/835afa01-3c31-468c-992a-375962c6c2d6-kube-api-access-2kbhh\") pod \"perf-node-gather-daemonset-t4d47\" (UID: \"835afa01-3c31-468c-992a-375962c6c2d6\") " pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" Jul 03 10:47:15.557797 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.557671 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/835afa01-3c31-468c-992a-375962c6c2d6-lib-modules\") pod \"perf-node-gather-daemonset-t4d47\" (UID: \"835afa01-3c31-468c-992a-375962c6c2d6\") " pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" Jul 03 10:47:15.557797 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.557741 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/835afa01-3c31-468c-992a-375962c6c2d6-sys\") pod \"perf-node-gather-daemonset-t4d47\" (UID: \"835afa01-3c31-468c-992a-375962c6c2d6\") " pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" Jul 03 10:47:15.557797 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.557767 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/835afa01-3c31-468c-992a-375962c6c2d6-podres\") pod \"perf-node-gather-daemonset-t4d47\" (UID: \"835afa01-3c31-468c-992a-375962c6c2d6\") " pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" Jul 03 10:47:15.557797 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.557776 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/835afa01-3c31-468c-992a-375962c6c2d6-proc\") pod \"perf-node-gather-daemonset-t4d47\" (UID: \"835afa01-3c31-468c-992a-375962c6c2d6\") " pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" Jul 03 10:47:15.557981 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.557831 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/835afa01-3c31-468c-992a-375962c6c2d6-proc\") pod \"perf-node-gather-daemonset-t4d47\" (UID: \"835afa01-3c31-468c-992a-375962c6c2d6\") " pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" Jul 03 10:47:15.557981 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.557837 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/835afa01-3c31-468c-992a-375962c6c2d6-lib-modules\") pod \"perf-node-gather-daemonset-t4d47\" (UID: \"835afa01-3c31-468c-992a-375962c6c2d6\") " pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" Jul 03 10:47:15.557981 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.557832 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/835afa01-3c31-468c-992a-375962c6c2d6-sys\") pod \"perf-node-gather-daemonset-t4d47\" (UID: \"835afa01-3c31-468c-992a-375962c6c2d6\") " pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" Jul 03 10:47:15.565144 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.565120 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kbhh\" (UniqueName: \"kubernetes.io/projected/835afa01-3c31-468c-992a-375962c6c2d6-kube-api-access-2kbhh\") pod \"perf-node-gather-daemonset-t4d47\" (UID: \"835afa01-3c31-468c-992a-375962c6c2d6\") " pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" Jul 03 10:47:15.687653 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.687562 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" Jul 03 10:47:15.822093 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:15.822060 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47"] Jul 03 10:47:15.825437 ip-10-0-143-105 kubenswrapper[2579]: W0703 10:47:15.825407 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod835afa01_3c31_468c_992a_375962c6c2d6.slice/crio-489703c0bee7558476e0bc0f96bf722b71210173d6e8de83212c6b5985b4ad7a WatchSource:0}: Error finding container 489703c0bee7558476e0bc0f96bf722b71210173d6e8de83212c6b5985b4ad7a: Status 404 returned error can't find the container with id 489703c0bee7558476e0bc0f96bf722b71210173d6e8de83212c6b5985b4ad7a Jul 03 10:47:16.760955 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:16.760918 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" event={"ID":"835afa01-3c31-468c-992a-375962c6c2d6","Type":"ContainerStarted","Data":"25c9b6b49a6886133a41eca21063057bbaddbb826f7a7d9b9b04e7b44a8544d5"} Jul 03 10:47:16.761459 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:16.760976 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" event={"ID":"835afa01-3c31-468c-992a-375962c6c2d6","Type":"ContainerStarted","Data":"489703c0bee7558476e0bc0f96bf722b71210173d6e8de83212c6b5985b4ad7a"} Jul 03 10:47:16.761459 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:16.761024 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" Jul 03 10:47:16.777605 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:16.777555 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" podStartSLOduration=1.777539821 podStartE2EDuration="1.777539821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 10:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 10:47:16.775864128 +0000 UTC m=+2507.814010740" watchObservedRunningTime="2026-07-03 10:47:16.777539821 +0000 UTC m=+2507.815686427" Jul 03 10:47:17.897361 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:17.897335 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-78277_c6b0923e-598e-4d07-bbd0-cdf55b61a8af/dns/0.log" Jul 03 10:47:17.920624 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:17.920598 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-78277_c6b0923e-598e-4d07-bbd0-cdf55b61a8af/kube-rbac-proxy/0.log" Jul 03 10:47:18.039607 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:18.039578 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4hpsh_f8254340-be1e-449d-8022-1d4da25997ff/dns-node-resolver/0.log" Jul 03 10:47:18.588005 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:18.587979 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-cdf799895-6vjq7_5f586edf-f0d1-45f9-9f55-330eed0dc1a4/registry/0.log" Jul 03 10:47:18.657512 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:18.657483 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vbnml_85cda35a-33a5-4f89-96f4-a2b3f8b585f7/node-ca/0.log" Jul 03 10:47:19.863749 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:19.863720 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b56494769-gmjkj_170f0891-9d61-4a9e-9211-4fcae92ddc58/kube-auth-proxy/0.log" Jul 03 10:47:20.843312 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:20.843287 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qz8nn_30ca9240-21bd-4d28-85d7-5ac0d2f438bb/serve-healthcheck-canary/0.log" Jul 03 10:47:21.435651 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:21.435627 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6cgf4_5570057e-5c51-449d-8390-7879cae4f8c8/kube-rbac-proxy/0.log" Jul 03 10:47:21.456078 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:21.456052 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6cgf4_5570057e-5c51-449d-8390-7879cae4f8c8/exporter/0.log" Jul 03 10:47:21.476051 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:21.476023 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6cgf4_5570057e-5c51-449d-8390-7879cae4f8c8/extractor/0.log" Jul 03 10:47:22.775282 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:22.775255 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dwvz5/perf-node-gather-daemonset-t4d47" Jul 03 10:47:23.748525 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:23.748495 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-s498m_e3467416-d3c8-461e-a5a2-a913adb30885/manager/0.log" Jul 03 10:47:23.868304 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:23.868273 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-hpnsx_ce204869-bbec-4287-9139-a6c80de9fa81/manager/0.log" Jul 03 10:47:24.147660 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:24.147635 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-w9t6l_2bee4600-2b04-4cbd-b3fe-518f0ee736bf/server/0.log" Jul 03 10:47:24.210282 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:24.210252 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-r9nqc_c3f3093f-95db-461e-99ae-f8d3153e85bb/manager/0.log" Jul 03 10:47:24.329319 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:24.329233 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789547c8f-5trlf_07f9e9d4-65d3-4b97-a4eb-a88c47a707f8/manager/0.log" Jul 03 10:47:24.346011 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:24.345991 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-789547c8f-5trlf_07f9e9d4-65d3-4b97-a4eb-a88c47a707f8/copy-manifests/0.log" Jul 03 10:47:26.136782 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:26.136758 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b56494769-gmjkj_170f0891-9d61-4a9e-9211-4fcae92ddc58/kube-auth-proxy/0.log" Jul 03 10:47:27.705784 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:27.705753 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b56494769-gmjkj_170f0891-9d61-4a9e-9211-4fcae92ddc58/kube-auth-proxy/0.log" Jul 03 10:47:30.791390 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:30.791360 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-h7ks5_f3f161fb-f33b-4aab-aa23-3a42315d24b0/migrator/0.log" Jul 03 10:47:30.811034 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:30.811010 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-h7ks5_f3f161fb-f33b-4aab-aa23-3a42315d24b0/graceful-termination/0.log" Jul 03 10:47:32.368299 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:32.368270 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dbhqx_21ff74d6-895a-4a92-9da8-b13c8db95a92/kube-multus-additional-cni-plugins/0.log" Jul 03 10:47:32.388154 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:32.388127 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dbhqx_21ff74d6-895a-4a92-9da8-b13c8db95a92/egress-router-binary-copy/0.log" Jul 03 10:47:32.411653 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:32.411629 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dbhqx_21ff74d6-895a-4a92-9da8-b13c8db95a92/cni-plugins/0.log" Jul 03 10:47:32.433010 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:32.432988 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dbhqx_21ff74d6-895a-4a92-9da8-b13c8db95a92/bond-cni-plugin/0.log" Jul 03 10:47:32.453369 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:32.453342 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dbhqx_21ff74d6-895a-4a92-9da8-b13c8db95a92/routeoverride-cni/0.log" Jul 03 10:47:32.472856 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:32.472835 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dbhqx_21ff74d6-895a-4a92-9da8-b13c8db95a92/whereabouts-cni-bincopy/0.log" Jul 03 10:47:32.492339 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:32.492315 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dbhqx_21ff74d6-895a-4a92-9da8-b13c8db95a92/whereabouts-cni/0.log" Jul 03 10:47:32.678921 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:32.678846 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fbl2t_94799f7a-6383-48ca-8ba2-6d5d0eee3148/kube-multus/0.log" Jul 03 10:47:32.858658 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:32.858628 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lwkn9_de5822f2-b52a-4283-938f-0395efeae942/network-metrics-daemon/0.log" Jul 03 10:47:32.878061 ip-10-0-143-105 kubenswrapper[2579]: I0703 10:47:32.878038 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lwkn9_de5822f2-b52a-4283-938f-0395efeae942/kube-rbac-proxy/0.log"