Jul 02 19:05:52.890845 ip-10-0-142-165 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 02 19:05:52.890855 ip-10-0-142-165 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 02 19:05:52.890862 ip-10-0-142-165 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 02 19:05:52.891085 ip-10-0-142-165 systemd[1]: Failed to start Kubernetes Kubelet. Jul 02 19:06:03.010804 ip-10-0-142-165 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 02 19:06:03.010824 ip-10-0-142-165 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 84b4a1f5a2fa4856bce5420eaa0a839a -- Jul 02 19:07:39.588255 ip-10-0-142-165 systemd[1]: Starting Kubernetes Kubelet... Jul 02 19:07:40.010061 ip-10-0-142-165 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 19:07:40.010061 ip-10-0-142-165 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 02 19:07:40.010061 ip-10-0-142-165 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 19:07:40.010061 ip-10-0-142-165 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 02 19:07:40.010061 ip-10-0-142-165 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 19:07:40.011178 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.011072 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 02 19:07:40.017676 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017655 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:07:40.017676 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017675 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:07:40.017676 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017680 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:07:40.017676 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017683 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:07:40.017840 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017688 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:07:40.017840 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017692 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:07:40.017840 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017695 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:07:40.017840 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017698 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:07:40.017840 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017701 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:07:40.017840 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017705 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:07:40.017840 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017708 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:07:40.017840 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017711 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:07:40.017840 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017714 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:07:40.017840 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017717 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:07:40.017840 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017720 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:07:40.017840 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017723 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:07:40.017840 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017733 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:07:40.017840 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017737 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:07:40.017840 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017739 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:07:40.017840 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017742 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:07:40.017840 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017745 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:07:40.017840 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017748 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:07:40.017840 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017751 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017753 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017756 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017759 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017762 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017764 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017767 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017771 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017774 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017777 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017779 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017782 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017785 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017787 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017789 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017793 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017795 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017798 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017800 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017803 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:07:40.018302 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017806 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017808 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017811 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017813 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017816 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017819 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017821 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017831 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017834 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017836 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017838 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017842 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017844 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017847 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017850 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017852 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017855 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017858 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017861 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017863 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:07:40.018793 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017866 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017869 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017871 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017874 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017876 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017879 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017882 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017885 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017887 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017890 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017893 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017895 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017897 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017900 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017904 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017907 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017909 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017912 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017914 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017917 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:07:40.019297 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017925 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:07:40.020094 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017942 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:07:40.020094 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017945 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:07:40.020094 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017948 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:07:40.020094 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017954 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:07:40.020094 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017957 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:07:40.020094 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017961 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:07:40.020094 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017964 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:07:40.020094 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017966 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:07:40.020094 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017969 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:07:40.020094 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017971 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:07:40.020094 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017974 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:07:40.020094 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017977 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:07:40.020094 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017980 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:07:40.020094 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017982 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:07:40.020094 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017985 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:07:40.020094 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.017987 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:07:40.020094 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018753 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:07:40.020094 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018760 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:07:40.020094 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018763 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:07:40.020761 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018766 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:07:40.020761 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018768 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:07:40.020761 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018771 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:07:40.020761 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018774 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:07:40.020761 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018779 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:07:40.020761 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018783 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:07:40.020761 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018786 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:07:40.020761 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018789 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:07:40.020761 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018792 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:07:40.020761 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018795 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:07:40.020761 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018798 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:07:40.020761 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018800 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:07:40.020761 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018803 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:07:40.020761 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018807 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:07:40.020761 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018810 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:07:40.020761 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018813 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:07:40.020761 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018815 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:07:40.020761 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018818 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:07:40.020761 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018821 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018824 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018826 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018829 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018831 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018834 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018837 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018839 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018842 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018844 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018847 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018849 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018852 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018855 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018858 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018860 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018863 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018866 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018868 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018871 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:07:40.021336 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018873 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:07:40.021820 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018877 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:07:40.021820 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018879 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:07:40.021820 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018882 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:07:40.021820 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018885 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:07:40.021820 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018887 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:07:40.021820 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018889 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:07:40.021820 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018893 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:07:40.021820 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018897 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:07:40.021820 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018900 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:07:40.021820 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018902 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:07:40.021820 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018905 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:07:40.021820 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018907 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:07:40.021820 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018910 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:07:40.021820 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018914 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:07:40.021820 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018917 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:07:40.021820 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018920 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:07:40.021820 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018923 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:07:40.021820 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018925 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:07:40.021820 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018945 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018948 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018951 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018954 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018957 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018960 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018963 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018965 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018968 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018971 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018974 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018978 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018983 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018986 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018988 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018991 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018993 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018996 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.018999 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019001 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:07:40.022303 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019004 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019007 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019010 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019013 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019015 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019018 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019021 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019023 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019026 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019028 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019031 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019034 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019036 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019039 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019041 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019044 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019046 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019049 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.019052 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021243 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:07:40.022780 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021253 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021256 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021259 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021262 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021265 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021268 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021271 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021274 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021276 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021279 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021282 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021285 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021288 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021291 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021293 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021296 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021299 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021301 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021304 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021306 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:07:40.023279 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021309 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021311 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021314 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021316 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021319 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021321 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021324 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021326 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021329 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021332 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021335 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021343 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021346 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021349 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021352 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021354 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021357 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021360 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021363 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021365 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:07:40.023766 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021368 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:07:40.024274 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021370 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:07:40.024274 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021373 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:07:40.024274 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021375 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:07:40.024274 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021378 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:07:40.024274 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021381 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:07:40.024274 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021384 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:07:40.024274 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021386 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:07:40.024274 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021389 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:07:40.024274 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021392 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:07:40.024274 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021398 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:07:40.024274 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021401 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:07:40.024274 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021405 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:07:40.024274 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021407 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:07:40.024274 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021411 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:07:40.024274 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021413 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:07:40.024274 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021416 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:07:40.024274 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021418 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:07:40.024274 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021421 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:07:40.024274 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021423 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021426 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021428 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021431 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021434 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021436 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021439 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021442 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021444 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021447 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021450 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021453 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021455 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021457 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021460 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021462 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021465 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021469 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021471 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021474 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:07:40.024724 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021476 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021479 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021481 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021486 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021489 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021492 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021494 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021497 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021501 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021504 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021507 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021510 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021512 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021515 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021517 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021520 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021523 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021526 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.021529 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021618 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 02 19:07:40.025213 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021627 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021634 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021639 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021644 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021647 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021652 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021656 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021659 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021662 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021666 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021670 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021673 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021676 2565 flags.go:64] FLAG: --cgroup-root="" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021679 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021682 2565 flags.go:64] FLAG: --client-ca-file="" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021685 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021689 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021693 2565 flags.go:64] FLAG: --cluster-domain="" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021697 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021701 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021704 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021708 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021718 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 02 19:07:40.025691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021721 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021725 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021728 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021732 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021735 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021738 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021741 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021744 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021748 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021751 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021754 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021757 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021760 2565 flags.go:64] FLAG: --enable-server="true" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021763 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021767 2565 flags.go:64] FLAG: --event-burst="100" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021770 2565 flags.go:64] FLAG: --event-qps="50" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021773 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021776 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021779 2565 flags.go:64] FLAG: --eviction-hard="" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021783 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021786 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021789 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021792 2565 flags.go:64] FLAG: --eviction-soft="" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021795 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021801 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 02 19:07:40.026281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021806 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021809 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021812 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021815 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021818 2565 flags.go:64] FLAG: --feature-gates="" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021822 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021825 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021829 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021832 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021835 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021838 2565 flags.go:64] FLAG: --help="false" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021841 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-142-165.ec2.internal" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021844 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021847 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021850 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021854 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021857 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021861 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021864 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021867 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021869 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021872 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021876 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021879 2565 flags.go:64] FLAG: --kube-reserved="" Jul 02 19:07:40.026886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021882 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021884 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021887 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021890 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021893 2565 flags.go:64] FLAG: --lock-file="" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021896 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021899 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021903 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021910 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021913 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021917 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021920 2565 flags.go:64] FLAG: --logging-format="text" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021922 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021939 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021943 2565 flags.go:64] FLAG: --manifest-url="" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021951 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021960 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021963 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021967 2565 flags.go:64] FLAG: --max-pods="110" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021970 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021974 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021977 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021980 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021983 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021986 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 02 19:07:40.027486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021989 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.021997 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022000 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022003 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022006 2565 flags.go:64] FLAG: --pod-cidr="" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022009 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022015 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022018 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022022 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022024 2565 flags.go:64] FLAG: --port="10250" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022028 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022030 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09370c48afae5ae70" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022033 2565 flags.go:64] FLAG: --qos-reserved="" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022036 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022040 2565 flags.go:64] FLAG: --register-node="true" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022044 2565 flags.go:64] FLAG: --register-with-taints="" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022048 2565 flags.go:64] FLAG: --registry-burst="10" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022051 2565 flags.go:64] FLAG: --registry-qps="5" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022054 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022057 2565 flags.go:64] FLAG: --reserved-memory="" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022061 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022064 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022067 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022070 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022073 2565 flags.go:64] FLAG: --runonce="false" Jul 02 19:07:40.028202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022075 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022079 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022082 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022084 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022088 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022091 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022094 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022097 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022100 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022103 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022106 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022109 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022112 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022115 2565 flags.go:64] FLAG: --system-cgroups="" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022118 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022123 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022126 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022129 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022133 2565 flags.go:64] FLAG: --tls-min-version="" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022136 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022139 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022142 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022146 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022150 2565 flags.go:64] FLAG: --v="2" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022154 2565 flags.go:64] FLAG: --version="false" Jul 02 19:07:40.028803 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022158 2565 flags.go:64] FLAG: --vmodule="" Jul 02 19:07:40.029459 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022162 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 02 19:07:40.029459 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.022166 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 02 19:07:40.029459 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022285 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:07:40.029459 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022289 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:07:40.029459 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022292 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:07:40.029459 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022295 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:07:40.029459 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022298 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:07:40.029459 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022300 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:07:40.029459 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022303 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:07:40.029459 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022306 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:07:40.029459 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022308 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:07:40.029459 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022311 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:07:40.029459 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022314 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:07:40.029459 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022317 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:07:40.029459 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022320 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:07:40.029459 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022322 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:07:40.029459 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022325 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:07:40.029459 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022328 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:07:40.029459 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022331 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:07:40.029957 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022334 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:07:40.029957 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022337 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:07:40.029957 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022339 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:07:40.029957 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022342 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:07:40.029957 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022345 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:07:40.029957 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022348 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:07:40.029957 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022351 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:07:40.029957 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022353 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:07:40.029957 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022356 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:07:40.029957 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022359 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:07:40.029957 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022361 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:07:40.029957 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022364 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:07:40.029957 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022366 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:07:40.029957 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022370 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:07:40.029957 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022374 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:07:40.029957 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022377 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:07:40.029957 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022380 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:07:40.029957 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022382 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:07:40.029957 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022385 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:07:40.030419 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022387 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:07:40.030419 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022390 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:07:40.030419 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022392 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:07:40.030419 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022395 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:07:40.030419 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022397 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:07:40.030419 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022400 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:07:40.030419 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022402 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:07:40.030419 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022404 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:07:40.030419 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022407 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:07:40.030419 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022410 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:07:40.030419 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022412 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:07:40.030419 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022414 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:07:40.030419 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022417 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:07:40.030419 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022421 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:07:40.030419 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022423 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:07:40.030419 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022426 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:07:40.030419 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022428 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:07:40.030419 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022432 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:07:40.030419 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022435 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022438 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022441 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022444 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022446 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022449 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022451 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022454 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022457 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022459 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022462 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022464 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022467 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022469 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022472 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022474 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022477 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022479 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022482 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022484 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:07:40.030884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022487 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022489 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022492 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022494 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022497 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022500 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022504 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022508 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022511 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022514 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022517 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022520 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022523 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022526 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022528 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022531 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022533 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022536 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022539 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022541 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:07:40.031382 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022544 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:07:40.031886 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022546 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:07:40.031886 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022549 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:07:40.031886 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022552 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:07:40.031886 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.022555 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:07:40.031886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.023309 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 19:07:40.031886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.030380 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 02 19:07:40.031886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.030397 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 02 19:07:40.031886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.030418 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 02 19:07:40.031886 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.030424 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 02 19:07:40.031886 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030484 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:07:40.031886 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030489 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:07:40.031886 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030492 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:07:40.031886 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030495 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:07:40.031886 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030498 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:07:40.031886 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030501 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:07:40.032286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030504 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:07:40.032286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030507 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:07:40.032286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030510 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:07:40.032286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030512 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:07:40.032286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030515 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:07:40.032286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030517 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:07:40.032286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030520 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:07:40.032286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030523 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:07:40.032286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030526 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:07:40.032286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030529 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:07:40.032286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030531 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:07:40.032286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030534 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:07:40.032286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030537 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:07:40.032286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030539 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:07:40.032286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030541 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:07:40.032286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030544 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:07:40.032286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030547 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:07:40.032286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030549 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:07:40.032286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030553 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030557 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030560 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030563 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030566 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030571 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030573 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030576 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030579 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030581 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030584 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030587 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030589 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030592 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030594 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030597 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030600 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030602 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030605 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030607 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:07:40.032743 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030610 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030612 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030615 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030617 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030620 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030622 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030624 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030627 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030630 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030632 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030634 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030637 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030639 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030642 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030644 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030647 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030651 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030654 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030657 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030659 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:07:40.033239 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030662 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:07:40.033718 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030666 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:07:40.033718 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030669 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:07:40.033718 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030672 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:07:40.033718 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030675 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:07:40.033718 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030677 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:07:40.033718 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030680 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:07:40.033718 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030682 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:07:40.033718 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030686 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:07:40.033718 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030690 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:07:40.033718 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030693 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:07:40.033718 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030696 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:07:40.033718 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030698 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:07:40.033718 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030701 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:07:40.033718 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030703 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:07:40.033718 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030706 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:07:40.033718 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030709 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:07:40.033718 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030711 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:07:40.033718 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030714 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:07:40.033718 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030716 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:07:40.034214 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030719 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:07:40.034214 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030721 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:07:40.034214 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030723 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:07:40.034214 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030726 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:07:40.034214 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030729 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:07:40.034214 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030731 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:07:40.034214 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030734 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:07:40.034214 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030736 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:07:40.034214 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030739 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:07:40.034214 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030742 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:07:40.034214 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030745 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:07:40.034214 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030748 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:07:40.034214 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030750 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:07:40.034214 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030753 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:07:40.034214 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030756 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:07:40.034599 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.030761 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 19:07:40.034599 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030876 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:07:40.034599 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030882 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:07:40.034599 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030885 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:07:40.034599 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030888 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:07:40.034599 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030891 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:07:40.034599 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030894 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:07:40.034599 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030897 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:07:40.034599 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030900 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:07:40.034599 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030903 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:07:40.034599 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030905 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:07:40.034599 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030908 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:07:40.034599 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030911 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:07:40.034599 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030914 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:07:40.034599 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030916 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030919 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030921 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030924 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030926 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030945 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030948 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030951 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030954 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030957 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030960 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030962 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030965 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030968 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030971 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030975 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030978 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030981 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030983 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030986 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:07:40.034990 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030989 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030991 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030994 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030997 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.030999 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031002 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031004 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031007 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031009 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031012 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031014 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031017 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031019 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031022 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031024 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031027 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031029 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031033 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031037 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031040 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:07:40.035489 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031042 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:07:40.036015 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031045 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:07:40.036015 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031047 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:07:40.036015 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031052 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:07:40.036015 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031055 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:07:40.036015 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031058 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:07:40.036015 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031061 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:07:40.036015 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031065 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:07:40.036015 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031069 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:07:40.036015 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031072 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:07:40.036015 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031074 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:07:40.036015 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031077 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:07:40.036015 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031080 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:07:40.036015 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031083 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:07:40.036015 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031085 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:07:40.036015 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031088 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:07:40.036015 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031090 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:07:40.036015 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031093 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:07:40.036015 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031095 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:07:40.036015 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031098 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031100 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031102 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031105 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031107 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031110 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031113 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031115 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031118 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031120 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031123 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031125 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031128 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031130 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031133 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031136 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031138 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031141 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031143 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031146 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:07:40.036471 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031149 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:07:40.037021 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031151 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:07:40.037021 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031155 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:07:40.037021 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031158 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:07:40.037021 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031160 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:07:40.037021 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031163 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:07:40.037021 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:40.031166 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:07:40.037021 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.031171 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 19:07:40.037021 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.031301 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 02 19:07:40.037021 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.033607 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 02 19:07:40.037021 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.035255 2565 server.go:1019] "Starting client certificate rotation" Jul 02 19:07:40.037021 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.035367 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 19:07:40.037021 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.036177 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 19:07:40.038129 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.038112 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 19:07:40.040978 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.040959 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 19:07:40.055647 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.055626 2565 log.go:25] "Validated CRI v1 runtime API" Jul 02 19:07:40.058162 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.058139 2565 log.go:25] "Validated CRI v1 image API" Jul 02 19:07:40.059000 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.058981 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 02 19:07:40.061708 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.061686 2565 fs.go:135] Filesystem UUIDs: map[5b291a49-e82a-42a7-b2ca-7b4f16d2eed0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9fd91c40-4298-4561-8512-3a92af3a7560:/dev/nvme0n1p4] Jul 02 19:07:40.061780 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.061707 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 02 19:07:40.064610 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.064589 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 19:07:40.067194 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.067078 2565 manager.go:216] Machine: {Timestamp:2026-07-02 19:07:40.065563252 +0000 UTC m=+0.360331133 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100135 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bb14d4687181e62592709f866b7fc SystemUUID:ec2bb14d-4687-181e-6259-2709f866b7fc BootID:84b4a1f5-a2fa-4856-bce5-420eaa0a839a Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f6:25:bc:1e:f7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f6:25:bc:1e:f7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:9a:c9:7a:bc:c4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 02 19:07:40.067194 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.067186 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 02 19:07:40.067340 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.067269 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 02 19:07:40.071068 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.071027 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 02 19:07:40.071217 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.071071 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-165.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 02 19:07:40.072659 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.072646 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 02 19:07:40.072659 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.072662 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 02 19:07:40.072740 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.072670 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 19:07:40.073451 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.073439 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 19:07:40.075332 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.075310 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 02 19:07:40.079635 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.079619 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 02 19:07:40.079798 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.079787 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 02 19:07:40.079831 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.079820 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 02 19:07:40.079831 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.079830 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 02 19:07:40.079892 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.079844 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 02 19:07:40.079892 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.079855 2565 kubelet.go:398] "Adding apiserver pod source" Jul 02 19:07:40.079892 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.079864 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 02 19:07:40.080310 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.080296 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 19:07:40.080374 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.080333 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 19:07:40.082216 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.082192 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wgklm" Jul 02 19:07:40.082864 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.082845 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 02 19:07:40.085516 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.085498 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 02 19:07:40.085590 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.085531 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 02 19:07:40.087038 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.087025 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 02 19:07:40.087123 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.087042 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 02 19:07:40.087123 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.087049 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 02 19:07:40.087123 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.087054 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 02 19:07:40.087123 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.087064 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 02 19:07:40.087123 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.087070 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 02 19:07:40.087123 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.087085 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 02 19:07:40.087123 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.087091 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 02 19:07:40.087123 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.087097 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 02 19:07:40.087123 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.087102 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 02 19:07:40.087123 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.087118 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 02 19:07:40.087123 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.087125 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 02 19:07:40.088854 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.088842 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 02 19:07:40.088854 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.088855 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 02 19:07:40.089549 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.089531 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wgklm" Jul 02 19:07:40.090641 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:40.090615 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-165.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 02 19:07:40.090723 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:40.090639 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 02 19:07:40.090876 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.090855 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-165.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 02 19:07:40.092632 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.092614 2565 server.go:1268] "Started kubelet" Jul 02 19:07:40.092715 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.092660 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 02 19:07:40.096667 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.092676 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 02 19:07:40.096812 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.096795 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 02 19:07:40.098117 ip-10-0-142-165 systemd[1]: Started Kubernetes Kubelet. Jul 02 19:07:40.101147 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:40.100120 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 02 19:07:40.101147 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.100519 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 02 19:07:40.101249 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.101168 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 02 19:07:40.107842 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.107820 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 02 19:07:40.107953 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.107825 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 02 19:07:40.108042 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:40.108029 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-165.ec2.internal\" not found" Jul 02 19:07:40.108658 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.108639 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 02 19:07:40.108805 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.108794 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 02 19:07:40.108866 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.108807 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 02 19:07:40.108923 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.108892 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 02 19:07:40.108923 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.108902 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 02 19:07:40.109085 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.109068 2565 factory.go:55] Registering systemd factory Jul 02 19:07:40.109167 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.109106 2565 factory.go:223] Registration of the systemd container factory successfully Jul 02 19:07:40.110217 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.110189 2565 factory.go:153] Registering CRI-O factory Jul 02 19:07:40.110217 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.110207 2565 factory.go:223] Registration of the crio container factory successfully Jul 02 19:07:40.110306 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.110257 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 02 19:07:40.110306 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.110279 2565 factory.go:103] Registering Raw factory Jul 02 19:07:40.110306 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.110293 2565 manager.go:1196] Started watching for new ooms in manager Jul 02 19:07:40.110397 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.110323 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:07:40.110663 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.110652 2565 manager.go:349] Starting recovery of all containers Jul 02 19:07:40.116232 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:40.116201 2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-165.ec2.internal\" not found" node="ip-10-0-142-165.ec2.internal" Jul 02 19:07:40.119888 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.119865 2565 manager.go:354] Recovery completed Jul 02 19:07:40.125552 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.125527 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:07:40.127781 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.127762 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:07:40.127866 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.127798 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:07:40.127866 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.127812 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:07:40.128423 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.128406 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 02 19:07:40.128423 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.128419 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:07:40.128599 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.128420 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 02 19:07:40.128599 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.128520 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 02 19:07:40.129693 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.129679 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:07:40.129844 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.129831 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:07:40.129901 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.129850 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:07:40.131557 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.131540 2565 policy_none.go:49] "None policy: Start" Jul 02 19:07:40.131858 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.131791 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 02 19:07:40.132001 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.131863 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 02 19:07:40.133750 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.133738 2565 policy_none.go:47] "Start" Jul 02 19:07:40.193802 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.193620 2565 manager.go:337] "Starting Device Plugin manager" Jul 02 19:07:40.193898 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:40.193851 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 02 19:07:40.193898 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.193865 2565 server.go:87] "Starting device plugin registration server" Jul 02 19:07:40.194189 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.194176 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 02 19:07:40.194236 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.194193 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 02 19:07:40.194312 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.194277 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 02 19:07:40.194364 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.194355 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 02 19:07:40.194364 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.194364 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 02 19:07:40.195251 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:40.195231 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 02 19:07:40.195404 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:40.195283 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-165.ec2.internal\" not found" Jul 02 19:07:40.212674 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.212630 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 02 19:07:40.214073 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.214056 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 02 19:07:40.214193 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.214101 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 02 19:07:40.214193 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.214128 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 02 19:07:40.214193 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:40.214175 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 02 19:07:40.216389 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.216357 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:07:40.294542 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.294438 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:07:40.295454 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.295439 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:07:40.295540 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.295468 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:07:40.295540 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.295477 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:07:40.295540 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.295501 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-165.ec2.internal" Jul 02 19:07:40.304568 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.304546 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-165.ec2.internal" Jul 02 19:07:40.304697 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:40.304575 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-165.ec2.internal\": node \"ip-10-0-142-165.ec2.internal\" not found" Jul 02 19:07:40.315201 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.315176 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-165.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal"] Jul 02 19:07:40.315330 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.315245 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:07:40.316728 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.316713 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:07:40.316800 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.316742 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:07:40.316800 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.316753 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:07:40.318560 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.318548 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:07:40.318666 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.318648 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-165.ec2.internal" Jul 02 19:07:40.318717 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.318687 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:07:40.319288 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.319272 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:07:40.319356 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.319298 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:07:40.319356 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.319307 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:07:40.319356 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.319275 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:07:40.319465 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.319368 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:07:40.319465 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.319378 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:07:40.320979 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.320967 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal" Jul 02 19:07:40.321029 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.320990 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:07:40.321630 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.321614 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:07:40.321700 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.321646 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:07:40.321700 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.321661 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:07:40.322193 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:40.322179 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-165.ec2.internal\" not found" Jul 02 19:07:40.338555 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:40.338535 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-165.ec2.internal\" not found" node="ip-10-0-142-165.ec2.internal" Jul 02 19:07:40.342515 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:40.342498 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-165.ec2.internal\" not found" node="ip-10-0-142-165.ec2.internal" Jul 02 19:07:40.410220 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.410188 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f74042b81dfcc4a8623a871af2883386-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal\" (UID: \"f74042b81dfcc4a8623a871af2883386\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal" Jul 02 19:07:40.410220 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.410219 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a9b387f1ea494ba8e3440d0784de7c5-config\") pod \"kube-apiserver-proxy-ip-10-0-142-165.ec2.internal\" (UID: \"7a9b387f1ea494ba8e3440d0784de7c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-165.ec2.internal" Jul 02 19:07:40.410406 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.410239 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f74042b81dfcc4a8623a871af2883386-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal\" (UID: \"f74042b81dfcc4a8623a871af2883386\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal" Jul 02 19:07:40.423182 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:40.423155 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-165.ec2.internal\" not found" Jul 02 19:07:40.510483 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.510447 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a9b387f1ea494ba8e3440d0784de7c5-config\") pod \"kube-apiserver-proxy-ip-10-0-142-165.ec2.internal\" (UID: \"7a9b387f1ea494ba8e3440d0784de7c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-165.ec2.internal" Jul 02 19:07:40.510483 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.510480 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f74042b81dfcc4a8623a871af2883386-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal\" (UID: \"f74042b81dfcc4a8623a871af2883386\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal" Jul 02 19:07:40.510676 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.510497 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f74042b81dfcc4a8623a871af2883386-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal\" (UID: \"f74042b81dfcc4a8623a871af2883386\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal" Jul 02 19:07:40.510676 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.510538 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f74042b81dfcc4a8623a871af2883386-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal\" (UID: \"f74042b81dfcc4a8623a871af2883386\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal" Jul 02 19:07:40.510676 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.510564 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7a9b387f1ea494ba8e3440d0784de7c5-config\") pod \"kube-apiserver-proxy-ip-10-0-142-165.ec2.internal\" (UID: \"7a9b387f1ea494ba8e3440d0784de7c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-165.ec2.internal" Jul 02 19:07:40.510676 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:40.510566 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f74042b81dfcc4a8623a871af2883386-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal\" (UID: \"f74042b81dfcc4a8623a871af2883386\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal" Jul 02 19:07:40.523601 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:40.523568 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-165.ec2.internal\" not found" Jul 02 19:07:41.803824 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:41.803772 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-165.ec2.internal\" not found" Jul 02 19:07:41.821984 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:41.821959 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-165.ec2.internal" Jul 02 19:07:41.825170 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:41.825151 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal" Jul 02 19:07:41.904439 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:41.904343 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-165.ec2.internal\" not found" Jul 02 19:07:42.004912 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:42.004871 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-165.ec2.internal\" not found" Jul 02 19:07:42.079494 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.079414 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:07:42.088419 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.088385 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal" Jul 02 19:07:42.100275 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.100247 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 19:07:42.102374 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.102343 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-165.ec2.internal" Jul 02 19:07:42.108941 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.108924 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 19:07:42.214424 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.214394 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 02 19:07:42.214908 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.214540 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 19:07:42.214908 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.214540 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 19:07:42.214908 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.214542 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 19:07:42.259745 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.259712 2565 apiserver.go:52] "Watching apiserver" Jul 02 19:07:42.274929 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.274436 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 02 19:07:42.276360 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.276322 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-hf8w8","openshift-multus/multus-additional-cni-plugins-dgwb4","openshift-network-diagnostics/network-check-target-m66fc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l","openshift-cluster-node-tuning-operator/tuned-l249r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal","openshift-multus/multus-mxvgp","openshift-multus/network-metrics-daemon-ld8fm","openshift-network-operator/iptables-alerter-swfnn","openshift-ovn-kubernetes/ovnkube-node-6bt54","kube-system/global-pull-secret-syncer-jfl78","kube-system/konnectivity-agent-x6shn","kube-system/kube-apiserver-proxy-ip-10-0-142-165.ec2.internal","openshift-dns/node-resolver-dqx74"] Jul 02 19:07:42.277539 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.277502 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 19:02:41 +0000 UTC" deadline="2028-02-22 00:39:07.534434467 +0000 UTC" Jul 02 19:07:42.277539 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.277537 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14381h31m25.25689979s" Jul 02 19:07:42.279238 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.279220 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.281627 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.281605 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 02 19:07:42.281721 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.281650 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 02 19:07:42.281721 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.281705 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 02 19:07:42.281817 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.281800 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 02 19:07:42.281867 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.281834 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g7gsx\"" Jul 02 19:07:42.281952 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.281929 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:42.282050 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:42.282003 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfl78" podUID="f714a87f-b1c3-4e57-8a2f-c1f61ab315be" Jul 02 19:07:42.284197 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.284170 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:07:42.284286 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:42.284225 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m66fc" podUID="b1b2b08c-8ac4-431a-92df-15eb9b42cc4b" Jul 02 19:07:42.286152 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.286133 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.287703 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.287686 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 02 19:07:42.288312 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.288295 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.288532 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.288509 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 02 19:07:42.288625 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.288531 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q25kw\"" Jul 02 19:07:42.288625 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.288539 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 02 19:07:42.288625 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.288602 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 02 19:07:42.290416 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.290378 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dqx74" Jul 02 19:07:42.290522 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.290510 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-28tn2\"" Jul 02 19:07:42.290698 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.290682 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 02 19:07:42.290765 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.290757 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:07:42.292485 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.292462 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-992l8\"" Jul 02 19:07:42.292588 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.292546 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-swfnn" Jul 02 19:07:42.292881 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.292865 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 02 19:07:42.292936 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.292891 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 02 19:07:42.294229 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294212 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-os-release\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.294307 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294240 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-etc-kubernetes\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.294307 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294267 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-kubelet-config\") pod \"global-pull-secret-syncer-jfl78\" (UID: \"f714a87f-b1c3-4e57-8a2f-c1f61ab315be\") " pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:42.294443 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294304 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhr4b\" (UniqueName: \"kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b\") pod \"network-check-target-m66fc\" (UID: \"b1b2b08c-8ac4-431a-92df-15eb9b42cc4b\") " pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:07:42.294443 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294342 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-run\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.294443 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294426 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-cnibin\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.294588 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294451 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f410247b-ac6f-4294-abf0-f34f1a57db0c-registration-dir\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.294588 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294473 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-host\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.294588 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294495 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-host-var-lib-cni-bin\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.294588 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294520 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm66w\" (UniqueName: \"kubernetes.io/projected/229973e8-3912-4f78-8d39-3f4a49ef2847-kube-api-access-gm66w\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.294588 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294540 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-dbus\") pod \"global-pull-secret-syncer-jfl78\" (UID: \"f714a87f-b1c3-4e57-8a2f-c1f61ab315be\") " pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:42.294588 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294559 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret\") pod \"global-pull-secret-syncer-jfl78\" (UID: \"f714a87f-b1c3-4e57-8a2f-c1f61ab315be\") " pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:42.294588 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294574 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f410247b-ac6f-4294-abf0-f34f1a57db0c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.294588 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294592 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f410247b-ac6f-4294-abf0-f34f1a57db0c-device-dir\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.295026 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294616 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f410247b-ac6f-4294-abf0-f34f1a57db0c-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.295026 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294667 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-sysconfig\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.295026 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294699 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-kubernetes\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.295026 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294739 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-systemd\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.295026 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294760 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:07:42.295026 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294779 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c758v\" (UniqueName: \"kubernetes.io/projected/744e479a-55a4-4de6-985b-a3b13ad4145f-kube-api-access-c758v\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.295026 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294809 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f410247b-ac6f-4294-abf0-f34f1a57db0c-socket-dir\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.295026 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:42.294828 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ld8fm" podUID="e21bc13b-a1b4-4934-9f3e-7eb280915f9b" Jul 02 19:07:42.295026 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294839 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-multus-socket-dir-parent\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.295026 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294869 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-multus-conf-dir\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.295026 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294897 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-sysctl-conf\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.295026 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294914 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-sys\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.295026 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294933 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-lib-modules\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.295026 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294950 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-var-lib-kubelet\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.295026 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.294995 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-host-run-k8s-cni-cncf-io\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.295026 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.295024 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/229973e8-3912-4f78-8d39-3f4a49ef2847-multus-daemon-config\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.295724 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.295055 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 02 19:07:42.295724 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.295065 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f410247b-ac6f-4294-abf0-f34f1a57db0c-sys-fs\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.295724 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.295104 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jt9b\" (UniqueName: \"kubernetes.io/projected/f410247b-ac6f-4294-abf0-f34f1a57db0c-kube-api-access-8jt9b\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.295724 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.295125 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/744e479a-55a4-4de6-985b-a3b13ad4145f-tmp\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.295724 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.295178 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-system-cni-dir\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.295724 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.295229 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-host-run-netns\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.295724 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.295276 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-hostroot\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.295724 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.295307 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-host-run-multus-certs\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.295724 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.295315 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 02 19:07:42.295724 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.295331 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-modprobe-d\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.295724 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.295371 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-sysctl-d\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.295724 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.295387 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-q4lpx\"" Jul 02 19:07:42.295724 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.295394 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/229973e8-3912-4f78-8d39-3f4a49ef2847-cni-binary-copy\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.295724 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.295417 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-host-var-lib-cni-multus\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.295724 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.295284 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:07:42.295724 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.295442 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-host-var-lib-kubelet\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.295724 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.295474 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-tuned\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.295724 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.295521 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-multus-cni-dir\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.300053 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.299982 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.300156 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.300091 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 19:07:42.302590 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.302572 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dt5td\"" Jul 02 19:07:42.302734 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.302712 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 02 19:07:42.302734 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.302732 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.302984 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.302923 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 02 19:07:42.305231 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.305214 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xd8wh\"" Jul 02 19:07:42.305231 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.305221 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 02 19:07:42.305394 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.305247 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 02 19:07:42.305499 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.305480 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 02 19:07:42.305600 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.305552 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 02 19:07:42.305600 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.305575 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 02 19:07:42.305711 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.305692 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hf8w8" Jul 02 19:07:42.306255 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.306238 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 02 19:07:42.308025 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.308009 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x6shn" Jul 02 19:07:42.308192 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.308165 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 02 19:07:42.308484 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.308469 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 02 19:07:42.308739 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.308714 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-69wfj\"" Jul 02 19:07:42.308967 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.308841 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 02 19:07:42.310413 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.310396 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wnwhj\"" Jul 02 19:07:42.310515 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.310428 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 02 19:07:42.310964 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.310847 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 02 19:07:42.323368 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.323325 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qskxl" Jul 02 19:07:42.332609 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.332528 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qskxl" Jul 02 19:07:42.389035 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.389005 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 02 19:07:42.395722 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.395681 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/229973e8-3912-4f78-8d39-3f4a49ef2847-cni-binary-copy\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.395722 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.395710 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-host-var-lib-cni-multus\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.395901 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.395748 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-host-var-lib-cni-multus\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.395901 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.395771 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-kubelet\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.395901 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.395787 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64cd41f8-0247-478f-bc3e-c16135f46632-ovnkube-script-lib\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.395901 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.395804 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-os-release\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.395901 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.395821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-kubelet-config\") pod \"global-pull-secret-syncer-jfl78\" (UID: \"f714a87f-b1c3-4e57-8a2f-c1f61ab315be\") " pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:42.395901 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.395860 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhr4b\" (UniqueName: \"kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b\") pod \"network-check-target-m66fc\" (UID: \"b1b2b08c-8ac4-431a-92df-15eb9b42cc4b\") " pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:07:42.396183 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.395919 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-os-release\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.396183 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.395919 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-kubelet-config\") pod \"global-pull-secret-syncer-jfl78\" (UID: \"f714a87f-b1c3-4e57-8a2f-c1f61ab315be\") " pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:42.396183 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.395945 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-cni-netd\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.396183 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.395980 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-os-release\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.396183 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396016 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-cnibin\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.396183 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396047 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-host\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.396183 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396069 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9bceedd-6e94-4988-baf5-5eb9eb417a7b-tmp-dir\") pod \"node-resolver-dqx74\" (UID: \"e9bceedd-6e94-4988-baf5-5eb9eb417a7b\") " pod="openshift-dns/node-resolver-dqx74" Jul 02 19:07:42.396183 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396075 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-cnibin\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.396183 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396091 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-var-lib-openvswitch\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.396183 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396135 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-host\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.396183 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396141 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-log-socket\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.396183 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396161 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64cd41f8-0247-478f-bc3e-c16135f46632-ovnkube-config\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.396183 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396176 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64cd41f8-0247-478f-bc3e-c16135f46632-env-overrides\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.396183 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396195 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-host-var-lib-cni-bin\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.396808 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396213 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm66w\" (UniqueName: \"kubernetes.io/projected/229973e8-3912-4f78-8d39-3f4a49ef2847-kube-api-access-gm66w\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.396808 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396220 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/229973e8-3912-4f78-8d39-3f4a49ef2847-cni-binary-copy\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.396808 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396254 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret\") pod \"global-pull-secret-syncer-jfl78\" (UID: \"f714a87f-b1c3-4e57-8a2f-c1f61ab315be\") " pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:42.396808 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396300 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-host-var-lib-cni-bin\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.396808 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:42.396335 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:07:42.396808 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:42.396413 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret podName:f714a87f-b1c3-4e57-8a2f-c1f61ab315be nodeName:}" failed. No retries permitted until 2026-07-02 19:07:42.896393152 +0000 UTC m=+2.011744358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret") pod "global-pull-secret-syncer-jfl78" (UID: "f714a87f-b1c3-4e57-8a2f-c1f61ab315be") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:07:42.396808 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396410 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-sysconfig\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.396808 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396333 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-sysconfig\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.396808 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396461 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-systemd\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.396808 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396494 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd9wv\" (UniqueName: \"kubernetes.io/projected/64cd41f8-0247-478f-bc3e-c16135f46632-kube-api-access-pd9wv\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.396808 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396523 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.396808 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396525 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-systemd\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.396808 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396560 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/315e9e92-0d3b-48f2-87d5-e3819e54fc0f-konnectivity-ca\") pod \"konnectivity-agent-x6shn\" (UID: \"315e9e92-0d3b-48f2-87d5-e3819e54fc0f\") " pod="kube-system/konnectivity-agent-x6shn" Jul 02 19:07:42.396808 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396587 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-node-log\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.396808 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396602 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-cni-bin\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.396808 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396619 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f410247b-ac6f-4294-abf0-f34f1a57db0c-socket-dir\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.396808 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396635 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-multus-socket-dir-parent\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.397490 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396660 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-lib-modules\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.397490 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-var-lib-kubelet\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.397490 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396696 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-etc-openvswitch\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.397490 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396715 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39-serviceca\") pod \"node-ca-hf8w8\" (UID: \"b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39\") " pod="openshift-image-registry/node-ca-hf8w8" Jul 02 19:07:42.397490 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396734 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b920bdb-1920-4df3-a4d3-cd710aec8492-iptables-alerter-script\") pod \"iptables-alerter-swfnn\" (UID: \"3b920bdb-1920-4df3-a4d3-cd710aec8492\") " pod="openshift-network-operator/iptables-alerter-swfnn" Jul 02 19:07:42.397490 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396749 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f410247b-ac6f-4294-abf0-f34f1a57db0c-socket-dir\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.397490 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396770 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-multus-socket-dir-parent\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.397490 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396772 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/229973e8-3912-4f78-8d39-3f4a49ef2847-multus-daemon-config\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.397490 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396788 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-var-lib-kubelet\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.397490 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396846 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-lib-modules\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.397490 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396859 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f410247b-ac6f-4294-abf0-f34f1a57db0c-sys-fs\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.397490 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396915 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f410247b-ac6f-4294-abf0-f34f1a57db0c-sys-fs\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.397490 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396954 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/744e479a-55a4-4de6-985b-a3b13ad4145f-tmp\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.397490 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.396996 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39-host\") pod \"node-ca-hf8w8\" (UID: \"b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39\") " pod="openshift-image-registry/node-ca-hf8w8" Jul 02 19:07:42.397490 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397022 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mmbk\" (UniqueName: \"kubernetes.io/projected/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-kube-api-access-5mmbk\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.397490 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397041 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-system-cni-dir\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.397490 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397060 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-host-run-netns\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.398228 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397078 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-hostroot\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.398228 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397094 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-host-run-multus-certs\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.398228 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397103 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-system-cni-dir\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.398228 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397109 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-modprobe-d\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.398228 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397114 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/229973e8-3912-4f78-8d39-3f4a49ef2847-multus-daemon-config\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.398228 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397138 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-sysctl-d\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.398228 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397158 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-host-run-multus-certs\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.398228 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397164 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-host-var-lib-kubelet\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.398228 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397160 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-hostroot\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.398228 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397189 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-tuned\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.398228 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397199 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-host-var-lib-kubelet\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.398228 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397161 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-host-run-netns\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.398228 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397206 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-modprobe-d\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.398228 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397235 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-sysctl-d\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.398228 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397243 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9bceedd-6e94-4988-baf5-5eb9eb417a7b-hosts-file\") pod \"node-resolver-dqx74\" (UID: \"e9bceedd-6e94-4988-baf5-5eb9eb417a7b\") " pod="openshift-dns/node-resolver-dqx74" Jul 02 19:07:42.398228 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397262 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 02 19:07:42.398228 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397304 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b920bdb-1920-4df3-a4d3-cd710aec8492-host-slash\") pod \"iptables-alerter-swfnn\" (UID: \"3b920bdb-1920-4df3-a4d3-cd710aec8492\") " pod="openshift-network-operator/iptables-alerter-swfnn" Jul 02 19:07:42.398228 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397328 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqrlv\" (UniqueName: \"kubernetes.io/projected/3b920bdb-1920-4df3-a4d3-cd710aec8492-kube-api-access-fqrlv\") pod \"iptables-alerter-swfnn\" (UID: \"3b920bdb-1920-4df3-a4d3-cd710aec8492\") " pod="openshift-network-operator/iptables-alerter-swfnn" Jul 02 19:07:42.398860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397373 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-cnibin\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.398860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397397 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-multus-cni-dir\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.398860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397418 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-etc-kubernetes\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.398860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397441 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-run\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.398860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397468 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-etc-kubernetes\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.398860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397462 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-run-openvswitch\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.398860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397504 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-run\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.398860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397511 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-multus-cni-dir\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.398860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397513 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.398860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397546 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.398860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397583 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f410247b-ac6f-4294-abf0-f34f1a57db0c-registration-dir\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.398860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397613 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-run-ovn-kubernetes\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.398860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397653 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f410247b-ac6f-4294-abf0-f34f1a57db0c-registration-dir\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.398860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397678 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6js9q\" (UniqueName: \"kubernetes.io/projected/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-kube-api-access-6js9q\") pod \"network-metrics-daemon-ld8fm\" (UID: \"e21bc13b-a1b4-4934-9f3e-7eb280915f9b\") " pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:07:42.398860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-dbus\") pod \"global-pull-secret-syncer-jfl78\" (UID: \"f714a87f-b1c3-4e57-8a2f-c1f61ab315be\") " pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:42.398860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397716 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f410247b-ac6f-4294-abf0-f34f1a57db0c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.398860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397760 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f410247b-ac6f-4294-abf0-f34f1a57db0c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.399329 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397771 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-dbus\") pod \"global-pull-secret-syncer-jfl78\" (UID: \"f714a87f-b1c3-4e57-8a2f-c1f61ab315be\") " pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:42.399329 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397773 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f410247b-ac6f-4294-abf0-f34f1a57db0c-device-dir\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.399329 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397804 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f410247b-ac6f-4294-abf0-f34f1a57db0c-device-dir\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.399329 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397805 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f410247b-ac6f-4294-abf0-f34f1a57db0c-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.399329 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397829 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-kubernetes\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.399329 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397855 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64cd41f8-0247-478f-bc3e-c16135f46632-ovn-node-metrics-cert\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.399329 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397882 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b57gg\" (UniqueName: \"kubernetes.io/projected/b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39-kube-api-access-b57gg\") pod \"node-ca-hf8w8\" (UID: \"b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39\") " pod="openshift-image-registry/node-ca-hf8w8" Jul 02 19:07:42.399329 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397890 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f410247b-ac6f-4294-abf0-f34f1a57db0c-etc-selinux\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.399329 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397907 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs\") pod \"network-metrics-daemon-ld8fm\" (UID: \"e21bc13b-a1b4-4934-9f3e-7eb280915f9b\") " pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:07:42.399329 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397915 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-kubernetes\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.399329 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397934 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c758v\" (UniqueName: \"kubernetes.io/projected/744e479a-55a4-4de6-985b-a3b13ad4145f-kube-api-access-c758v\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.399329 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.397959 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-run-netns\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.399329 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.398062 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-system-cni-dir\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.399329 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.398097 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.399329 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.398122 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.399329 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.398153 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-multus-conf-dir\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.399783 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.398178 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-sysctl-conf\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.399783 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.398202 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-sys\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.399783 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.398230 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-multus-conf-dir\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.399783 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.398241 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssnhz\" (UniqueName: \"kubernetes.io/projected/e9bceedd-6e94-4988-baf5-5eb9eb417a7b-kube-api-access-ssnhz\") pod \"node-resolver-dqx74\" (UID: \"e9bceedd-6e94-4988-baf5-5eb9eb417a7b\") " pod="openshift-dns/node-resolver-dqx74" Jul 02 19:07:42.399783 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.398263 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-run-ovn\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.399783 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.398290 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-sys\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.399783 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.398291 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-host-run-k8s-cni-cncf-io\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.399783 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.398316 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-sysctl-conf\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.399783 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.398327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jt9b\" (UniqueName: \"kubernetes.io/projected/f410247b-ac6f-4294-abf0-f34f1a57db0c-kube-api-access-8jt9b\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.399783 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.398330 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/229973e8-3912-4f78-8d39-3f4a49ef2847-host-run-k8s-cni-cncf-io\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.399783 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.398378 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-systemd-units\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.399783 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.398409 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-slash\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.399783 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.398437 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/315e9e92-0d3b-48f2-87d5-e3819e54fc0f-agent-certs\") pod \"konnectivity-agent-x6shn\" (UID: \"315e9e92-0d3b-48f2-87d5-e3819e54fc0f\") " pod="kube-system/konnectivity-agent-x6shn" Jul 02 19:07:42.399783 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.398458 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-run-systemd\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.400298 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.400278 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/744e479a-55a4-4de6-985b-a3b13ad4145f-etc-tuned\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.400329 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.400310 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/744e479a-55a4-4de6-985b-a3b13ad4145f-tmp\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.413605 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:42.413581 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:07:42.413753 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:42.413738 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:07:42.413753 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:42.413758 2565 projected.go:196] Error preparing data for projected volume kube-api-access-jhr4b for pod openshift-network-diagnostics/network-check-target-m66fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:07:42.413879 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:42.413835 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b podName:b1b2b08c-8ac4-431a-92df-15eb9b42cc4b nodeName:}" failed. No retries permitted until 2026-07-02 19:07:42.913810926 +0000 UTC m=+2.029162144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jhr4b" (UniqueName: "kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b") pod "network-check-target-m66fc" (UID: "b1b2b08c-8ac4-431a-92df-15eb9b42cc4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:07:42.416907 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.416882 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm66w\" (UniqueName: \"kubernetes.io/projected/229973e8-3912-4f78-8d39-3f4a49ef2847-kube-api-access-gm66w\") pod \"multus-mxvgp\" (UID: \"229973e8-3912-4f78-8d39-3f4a49ef2847\") " pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.417249 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.417227 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c758v\" (UniqueName: \"kubernetes.io/projected/744e479a-55a4-4de6-985b-a3b13ad4145f-kube-api-access-c758v\") pod \"tuned-l249r\" (UID: \"744e479a-55a4-4de6-985b-a3b13ad4145f\") " pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.417328 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.417252 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jt9b\" (UniqueName: \"kubernetes.io/projected/f410247b-ac6f-4294-abf0-f34f1a57db0c-kube-api-access-8jt9b\") pod \"aws-ebs-csi-driver-node-6qq7l\" (UID: \"f410247b-ac6f-4294-abf0-f34f1a57db0c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.436598 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:42.436566 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74042b81dfcc4a8623a871af2883386.slice/crio-dc58e880bc1a36349c3388431938e585eec03ab8e0837ec8701439be7bc2b9de WatchSource:0}: Error finding container dc58e880bc1a36349c3388431938e585eec03ab8e0837ec8701439be7bc2b9de: Status 404 returned error can't find the container with id dc58e880bc1a36349c3388431938e585eec03ab8e0837ec8701439be7bc2b9de Jul 02 19:07:42.436795 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:42.436779 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9b387f1ea494ba8e3440d0784de7c5.slice/crio-b9195850993ad890e7faa59dc3269a3fc6b01523a8747261bff5dad3630de63d WatchSource:0}: Error finding container b9195850993ad890e7faa59dc3269a3fc6b01523a8747261bff5dad3630de63d: Status 404 returned error can't find the container with id b9195850993ad890e7faa59dc3269a3fc6b01523a8747261bff5dad3630de63d Jul 02 19:07:42.441658 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.441640 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:07:42.499765 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.499720 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd9wv\" (UniqueName: \"kubernetes.io/projected/64cd41f8-0247-478f-bc3e-c16135f46632-kube-api-access-pd9wv\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.499765 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.499758 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.499990 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.499785 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/315e9e92-0d3b-48f2-87d5-e3819e54fc0f-konnectivity-ca\") pod \"konnectivity-agent-x6shn\" (UID: \"315e9e92-0d3b-48f2-87d5-e3819e54fc0f\") " pod="kube-system/konnectivity-agent-x6shn" Jul 02 19:07:42.499990 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.499804 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-node-log\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.499990 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.499829 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-cni-bin\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.499990 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.499855 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-etc-openvswitch\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.499990 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.499876 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39-serviceca\") pod \"node-ca-hf8w8\" (UID: \"b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39\") " pod="openshift-image-registry/node-ca-hf8w8" Jul 02 19:07:42.499990 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.499882 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-node-log\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.499990 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.499896 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b920bdb-1920-4df3-a4d3-cd710aec8492-iptables-alerter-script\") pod \"iptables-alerter-swfnn\" (UID: \"3b920bdb-1920-4df3-a4d3-cd710aec8492\") " pod="openshift-network-operator/iptables-alerter-swfnn" Jul 02 19:07:42.499990 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.499936 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39-host\") pod \"node-ca-hf8w8\" (UID: \"b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39\") " pod="openshift-image-registry/node-ca-hf8w8" Jul 02 19:07:42.499990 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.499939 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-cni-bin\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.499990 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.499958 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mmbk\" (UniqueName: \"kubernetes.io/projected/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-kube-api-access-5mmbk\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.500505 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500006 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9bceedd-6e94-4988-baf5-5eb9eb417a7b-hosts-file\") pod \"node-resolver-dqx74\" (UID: \"e9bceedd-6e94-4988-baf5-5eb9eb417a7b\") " pod="openshift-dns/node-resolver-dqx74" Jul 02 19:07:42.500505 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500419 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39-serviceca\") pod \"node-ca-hf8w8\" (UID: \"b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39\") " pod="openshift-image-registry/node-ca-hf8w8" Jul 02 19:07:42.500505 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500484 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.500505 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500486 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3b920bdb-1920-4df3-a4d3-cd710aec8492-iptables-alerter-script\") pod \"iptables-alerter-swfnn\" (UID: \"3b920bdb-1920-4df3-a4d3-cd710aec8492\") " pod="openshift-network-operator/iptables-alerter-swfnn" Jul 02 19:07:42.500642 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500543 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-etc-openvswitch\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.500642 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500589 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b920bdb-1920-4df3-a4d3-cd710aec8492-host-slash\") pod \"iptables-alerter-swfnn\" (UID: \"3b920bdb-1920-4df3-a4d3-cd710aec8492\") " pod="openshift-network-operator/iptables-alerter-swfnn" Jul 02 19:07:42.500642 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500610 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqrlv\" (UniqueName: \"kubernetes.io/projected/3b920bdb-1920-4df3-a4d3-cd710aec8492-kube-api-access-fqrlv\") pod \"iptables-alerter-swfnn\" (UID: \"3b920bdb-1920-4df3-a4d3-cd710aec8492\") " pod="openshift-network-operator/iptables-alerter-swfnn" Jul 02 19:07:42.500642 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500622 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/315e9e92-0d3b-48f2-87d5-e3819e54fc0f-konnectivity-ca\") pod \"konnectivity-agent-x6shn\" (UID: \"315e9e92-0d3b-48f2-87d5-e3819e54fc0f\") " pod="kube-system/konnectivity-agent-x6shn" Jul 02 19:07:42.500642 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500633 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-cnibin\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.500884 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500653 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-run-openvswitch\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.500884 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500673 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.500884 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500694 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.500884 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500699 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e9bceedd-6e94-4988-baf5-5eb9eb417a7b-hosts-file\") pod \"node-resolver-dqx74\" (UID: \"e9bceedd-6e94-4988-baf5-5eb9eb417a7b\") " pod="openshift-dns/node-resolver-dqx74" Jul 02 19:07:42.500884 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500728 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-run-ovn-kubernetes\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.500884 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500740 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-run-openvswitch\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.500884 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6js9q\" (UniqueName: \"kubernetes.io/projected/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-kube-api-access-6js9q\") pod \"network-metrics-daemon-ld8fm\" (UID: \"e21bc13b-a1b4-4934-9f3e-7eb280915f9b\") " pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:07:42.500884 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500753 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.500884 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500793 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b920bdb-1920-4df3-a4d3-cd710aec8492-host-slash\") pod \"iptables-alerter-swfnn\" (UID: \"3b920bdb-1920-4df3-a4d3-cd710aec8492\") " pod="openshift-network-operator/iptables-alerter-swfnn" Jul 02 19:07:42.500884 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500798 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39-host\") pod \"node-ca-hf8w8\" (UID: \"b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39\") " pod="openshift-image-registry/node-ca-hf8w8" Jul 02 19:07:42.500884 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500807 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-cnibin\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.500884 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500810 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64cd41f8-0247-478f-bc3e-c16135f46632-ovn-node-metrics-cert\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.500884 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500847 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b57gg\" (UniqueName: \"kubernetes.io/projected/b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39-kube-api-access-b57gg\") pod \"node-ca-hf8w8\" (UID: \"b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39\") " pod="openshift-image-registry/node-ca-hf8w8" Jul 02 19:07:42.500884 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500872 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs\") pod \"network-metrics-daemon-ld8fm\" (UID: \"e21bc13b-a1b4-4934-9f3e-7eb280915f9b\") " pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:07:42.500884 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500888 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-run-ovn-kubernetes\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.501606 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500896 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-run-netns\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.501606 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500920 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-system-cni-dir\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.501606 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500943 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.501606 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:42.500963 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:07:42.501606 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500967 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.501606 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500971 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-run-netns\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.501606 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.500998 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssnhz\" (UniqueName: \"kubernetes.io/projected/e9bceedd-6e94-4988-baf5-5eb9eb417a7b-kube-api-access-ssnhz\") pod \"node-resolver-dqx74\" (UID: \"e9bceedd-6e94-4988-baf5-5eb9eb417a7b\") " pod="openshift-dns/node-resolver-dqx74" Jul 02 19:07:42.501606 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501004 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-system-cni-dir\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.501606 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:42.501027 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs podName:e21bc13b-a1b4-4934-9f3e-7eb280915f9b nodeName:}" failed. No retries permitted until 2026-07-02 19:07:43.001000934 +0000 UTC m=+2.116352149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs") pod "network-metrics-daemon-ld8fm" (UID: "e21bc13b-a1b4-4934-9f3e-7eb280915f9b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:07:42.501606 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501063 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-run-ovn\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.501606 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501093 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-systemd-units\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.501606 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501118 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-slash\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.501606 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501162 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-run-ovn\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.501606 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501172 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/315e9e92-0d3b-48f2-87d5-e3819e54fc0f-agent-certs\") pod \"konnectivity-agent-x6shn\" (UID: \"315e9e92-0d3b-48f2-87d5-e3819e54fc0f\") " pod="kube-system/konnectivity-agent-x6shn" Jul 02 19:07:42.501606 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501169 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.501606 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501197 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-run-systemd\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.501606 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501215 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-slash\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.502415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501230 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-kubelet\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.502415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501240 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-run-systemd\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.502415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501254 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-systemd-units\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.502415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501254 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.502415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501255 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64cd41f8-0247-478f-bc3e-c16135f46632-ovnkube-script-lib\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.502415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501296 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-kubelet\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.502415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501310 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-cni-netd\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.502415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501336 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-os-release\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.502415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501379 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9bceedd-6e94-4988-baf5-5eb9eb417a7b-tmp-dir\") pod \"node-resolver-dqx74\" (UID: \"e9bceedd-6e94-4988-baf5-5eb9eb417a7b\") " pod="openshift-dns/node-resolver-dqx74" Jul 02 19:07:42.502415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501401 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-host-cni-netd\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.502415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501435 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-os-release\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.502415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501403 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-var-lib-openvswitch\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.502415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501462 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-var-lib-openvswitch\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.502415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501475 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.502415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501481 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-log-socket\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.502415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501510 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/64cd41f8-0247-478f-bc3e-c16135f46632-log-socket\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.502415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501529 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64cd41f8-0247-478f-bc3e-c16135f46632-ovnkube-config\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.503224 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501551 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64cd41f8-0247-478f-bc3e-c16135f46632-env-overrides\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.503224 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501683 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9bceedd-6e94-4988-baf5-5eb9eb417a7b-tmp-dir\") pod \"node-resolver-dqx74\" (UID: \"e9bceedd-6e94-4988-baf5-5eb9eb417a7b\") " pod="openshift-dns/node-resolver-dqx74" Jul 02 19:07:42.503224 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501867 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/64cd41f8-0247-478f-bc3e-c16135f46632-ovnkube-script-lib\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.503224 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501922 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/64cd41f8-0247-478f-bc3e-c16135f46632-env-overrides\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.503224 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.501984 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/64cd41f8-0247-478f-bc3e-c16135f46632-ovnkube-config\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.503398 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.503248 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/315e9e92-0d3b-48f2-87d5-e3819e54fc0f-agent-certs\") pod \"konnectivity-agent-x6shn\" (UID: \"315e9e92-0d3b-48f2-87d5-e3819e54fc0f\") " pod="kube-system/konnectivity-agent-x6shn" Jul 02 19:07:42.503437 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.503412 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/64cd41f8-0247-478f-bc3e-c16135f46632-ovn-node-metrics-cert\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.510386 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.510332 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mmbk\" (UniqueName: \"kubernetes.io/projected/ce6ec4d4-1f88-40e4-b020-e1e137efd53a-kube-api-access-5mmbk\") pod \"multus-additional-cni-plugins-dgwb4\" (UID: \"ce6ec4d4-1f88-40e4-b020-e1e137efd53a\") " pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.510659 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.510632 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b57gg\" (UniqueName: \"kubernetes.io/projected/b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39-kube-api-access-b57gg\") pod \"node-ca-hf8w8\" (UID: \"b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39\") " pod="openshift-image-registry/node-ca-hf8w8" Jul 02 19:07:42.510702 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.510638 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6js9q\" (UniqueName: \"kubernetes.io/projected/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-kube-api-access-6js9q\") pod \"network-metrics-daemon-ld8fm\" (UID: \"e21bc13b-a1b4-4934-9f3e-7eb280915f9b\") " pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:07:42.511252 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.511236 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqrlv\" (UniqueName: \"kubernetes.io/projected/3b920bdb-1920-4df3-a4d3-cd710aec8492-kube-api-access-fqrlv\") pod \"iptables-alerter-swfnn\" (UID: \"3b920bdb-1920-4df3-a4d3-cd710aec8492\") " pod="openshift-network-operator/iptables-alerter-swfnn" Jul 02 19:07:42.511775 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.511753 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd9wv\" (UniqueName: \"kubernetes.io/projected/64cd41f8-0247-478f-bc3e-c16135f46632-kube-api-access-pd9wv\") pod \"ovnkube-node-6bt54\" (UID: \"64cd41f8-0247-478f-bc3e-c16135f46632\") " pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.513583 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.513564 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssnhz\" (UniqueName: \"kubernetes.io/projected/e9bceedd-6e94-4988-baf5-5eb9eb417a7b-kube-api-access-ssnhz\") pod \"node-resolver-dqx74\" (UID: \"e9bceedd-6e94-4988-baf5-5eb9eb417a7b\") " pod="openshift-dns/node-resolver-dqx74" Jul 02 19:07:42.609235 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.609132 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mxvgp" Jul 02 19:07:42.619358 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:42.619322 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229973e8_3912_4f78_8d39_3f4a49ef2847.slice/crio-f1d612f8bf9c4d9570e205d0dd3e76a84965c0fc73f33371d13272ee8189534a WatchSource:0}: Error finding container f1d612f8bf9c4d9570e205d0dd3e76a84965c0fc73f33371d13272ee8189534a: Status 404 returned error can't find the container with id f1d612f8bf9c4d9570e205d0dd3e76a84965c0fc73f33371d13272ee8189534a Jul 02 19:07:42.625546 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.625525 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" Jul 02 19:07:42.634326 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:42.634292 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf410247b_ac6f_4294_abf0_f34f1a57db0c.slice/crio-442e38335f60e67e100a3b527abcc1fdca6c4ff1bdf848c1caf30ef8cf4801d6 WatchSource:0}: Error finding container 442e38335f60e67e100a3b527abcc1fdca6c4ff1bdf848c1caf30ef8cf4801d6: Status 404 returned error can't find the container with id 442e38335f60e67e100a3b527abcc1fdca6c4ff1bdf848c1caf30ef8cf4801d6 Jul 02 19:07:42.640584 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.640566 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-l249r" Jul 02 19:07:42.648589 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:42.648565 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744e479a_55a4_4de6_985b_a3b13ad4145f.slice/crio-210ad5785d2b546bc071488f7a42c7de7e96a9ca33afca267ab6c0b7189bf1fc WatchSource:0}: Error finding container 210ad5785d2b546bc071488f7a42c7de7e96a9ca33afca267ab6c0b7189bf1fc: Status 404 returned error can't find the container with id 210ad5785d2b546bc071488f7a42c7de7e96a9ca33afca267ab6c0b7189bf1fc Jul 02 19:07:42.670360 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.670322 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dqx74" Jul 02 19:07:42.678481 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:42.678452 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9bceedd_6e94_4988_baf5_5eb9eb417a7b.slice/crio-537ca8bd64a9ec4c92ef233e955eb71b0bf912b666d48eb6c24e92064c28be44 WatchSource:0}: Error finding container 537ca8bd64a9ec4c92ef233e955eb71b0bf912b666d48eb6c24e92064c28be44: Status 404 returned error can't find the container with id 537ca8bd64a9ec4c92ef233e955eb71b0bf912b666d48eb6c24e92064c28be44 Jul 02 19:07:42.684511 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.684490 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:07:42.685867 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.685849 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-swfnn" Jul 02 19:07:42.691914 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.691873 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dgwb4" Jul 02 19:07:42.694132 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:42.694103 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b920bdb_1920_4df3_a4d3_cd710aec8492.slice/crio-1c6bee76d3951d264c276303ea5b69fe870314c8f384b63116e3ea6245eda05d WatchSource:0}: Error finding container 1c6bee76d3951d264c276303ea5b69fe870314c8f384b63116e3ea6245eda05d: Status 404 returned error can't find the container with id 1c6bee76d3951d264c276303ea5b69fe870314c8f384b63116e3ea6245eda05d Jul 02 19:07:42.702137 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:42.702108 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6ec4d4_1f88_40e4_b020_e1e137efd53a.slice/crio-53714ce5fecc575a6ec30ca5e7ec6fa04604b0de37b3276eb82ee5fdd36a90fa WatchSource:0}: Error finding container 53714ce5fecc575a6ec30ca5e7ec6fa04604b0de37b3276eb82ee5fdd36a90fa: Status 404 returned error can't find the container with id 53714ce5fecc575a6ec30ca5e7ec6fa04604b0de37b3276eb82ee5fdd36a90fa Jul 02 19:07:42.712246 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.712224 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:07:42.721978 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:42.721954 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64cd41f8_0247_478f_bc3e_c16135f46632.slice/crio-532ae55c443dd66ae33c2e31ee5a3f7a37ad1fb8fa35ea6c9ddfdad8ed46e876 WatchSource:0}: Error finding container 532ae55c443dd66ae33c2e31ee5a3f7a37ad1fb8fa35ea6c9ddfdad8ed46e876: Status 404 returned error can't find the container with id 532ae55c443dd66ae33c2e31ee5a3f7a37ad1fb8fa35ea6c9ddfdad8ed46e876 Jul 02 19:07:42.737551 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.737522 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hf8w8" Jul 02 19:07:42.743739 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.743718 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x6shn" Jul 02 19:07:42.747154 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:42.747121 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b59a24_8f2e_4753_a3b2_fb43e2e6cd39.slice/crio-5de32f9fc24b6dd0d753b096b9c703dc6d1ee3b717d6231e9015405b159c1570 WatchSource:0}: Error finding container 5de32f9fc24b6dd0d753b096b9c703dc6d1ee3b717d6231e9015405b159c1570: Status 404 returned error can't find the container with id 5de32f9fc24b6dd0d753b096b9c703dc6d1ee3b717d6231e9015405b159c1570 Jul 02 19:07:42.751873 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:07:42.751851 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315e9e92_0d3b_48f2_87d5_e3819e54fc0f.slice/crio-dbaa4367131dcedb3e1e369305298648fafb5ea4fa4227e409b6d19542d069d6 WatchSource:0}: Error finding container dbaa4367131dcedb3e1e369305298648fafb5ea4fa4227e409b6d19542d069d6: Status 404 returned error can't find the container with id dbaa4367131dcedb3e1e369305298648fafb5ea4fa4227e409b6d19542d069d6 Jul 02 19:07:42.903495 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:42.903405 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret\") pod \"global-pull-secret-syncer-jfl78\" (UID: \"f714a87f-b1c3-4e57-8a2f-c1f61ab315be\") " pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:42.903663 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:42.903533 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:07:42.903663 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:42.903595 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret podName:f714a87f-b1c3-4e57-8a2f-c1f61ab315be nodeName:}" failed. No retries permitted until 2026-07-02 19:07:43.903581071 +0000 UTC m=+3.018932297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret") pod "global-pull-secret-syncer-jfl78" (UID: "f714a87f-b1c3-4e57-8a2f-c1f61ab315be") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:07:43.004139 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:43.003763 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs\") pod \"network-metrics-daemon-ld8fm\" (UID: \"e21bc13b-a1b4-4934-9f3e-7eb280915f9b\") " pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:07:43.004139 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:43.003832 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhr4b\" (UniqueName: \"kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b\") pod \"network-check-target-m66fc\" (UID: \"b1b2b08c-8ac4-431a-92df-15eb9b42cc4b\") " pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:07:43.004139 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:43.003920 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:07:43.004139 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:43.004000 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs podName:e21bc13b-a1b4-4934-9f3e-7eb280915f9b nodeName:}" failed. No retries permitted until 2026-07-02 19:07:44.003979613 +0000 UTC m=+3.119330829 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs") pod "network-metrics-daemon-ld8fm" (UID: "e21bc13b-a1b4-4934-9f3e-7eb280915f9b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:07:43.004139 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:43.004020 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:07:43.004139 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:43.004038 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:07:43.004139 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:43.004051 2565 projected.go:196] Error preparing data for projected volume kube-api-access-jhr4b for pod openshift-network-diagnostics/network-check-target-m66fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:07:43.004139 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:43.004106 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b podName:b1b2b08c-8ac4-431a-92df-15eb9b42cc4b nodeName:}" failed. No retries permitted until 2026-07-02 19:07:44.004088153 +0000 UTC m=+3.119439374 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jhr4b" (UniqueName: "kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b") pod "network-check-target-m66fc" (UID: "b1b2b08c-8ac4-431a-92df-15eb9b42cc4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:07:43.295059 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:43.294970 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:07:43.337592 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:43.337515 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 19:02:42 +0000 UTC" deadline="2028-01-09 18:46:46.452027907 +0000 UTC" Jul 02 19:07:43.337592 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:43.337550 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13343h39m3.114482994s" Jul 02 19:07:43.417251 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:43.417183 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x6shn" event={"ID":"315e9e92-0d3b-48f2-87d5-e3819e54fc0f","Type":"ContainerStarted","Data":"dbaa4367131dcedb3e1e369305298648fafb5ea4fa4227e409b6d19542d069d6"} Jul 02 19:07:43.428564 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:43.428524 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-swfnn" event={"ID":"3b920bdb-1920-4df3-a4d3-cd710aec8492","Type":"ContainerStarted","Data":"1c6bee76d3951d264c276303ea5b69fe870314c8f384b63116e3ea6245eda05d"} Jul 02 19:07:43.436422 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:43.436385 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" event={"ID":"f410247b-ac6f-4294-abf0-f34f1a57db0c","Type":"ContainerStarted","Data":"442e38335f60e67e100a3b527abcc1fdca6c4ff1bdf848c1caf30ef8cf4801d6"} Jul 02 19:07:43.453455 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:43.453416 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mxvgp" event={"ID":"229973e8-3912-4f78-8d39-3f4a49ef2847","Type":"ContainerStarted","Data":"f1d612f8bf9c4d9570e205d0dd3e76a84965c0fc73f33371d13272ee8189534a"} Jul 02 19:07:43.460376 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:43.460317 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal" event={"ID":"f74042b81dfcc4a8623a871af2883386","Type":"ContainerStarted","Data":"dc58e880bc1a36349c3388431938e585eec03ab8e0837ec8701439be7bc2b9de"} Jul 02 19:07:43.473227 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:43.473187 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-165.ec2.internal" event={"ID":"7a9b387f1ea494ba8e3440d0784de7c5","Type":"ContainerStarted","Data":"b9195850993ad890e7faa59dc3269a3fc6b01523a8747261bff5dad3630de63d"} Jul 02 19:07:43.494046 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:43.493953 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hf8w8" event={"ID":"b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39","Type":"ContainerStarted","Data":"5de32f9fc24b6dd0d753b096b9c703dc6d1ee3b717d6231e9015405b159c1570"} Jul 02 19:07:43.509657 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:43.509481 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" event={"ID":"64cd41f8-0247-478f-bc3e-c16135f46632","Type":"ContainerStarted","Data":"532ae55c443dd66ae33c2e31ee5a3f7a37ad1fb8fa35ea6c9ddfdad8ed46e876"} Jul 02 19:07:43.519625 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:43.519570 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgwb4" event={"ID":"ce6ec4d4-1f88-40e4-b020-e1e137efd53a","Type":"ContainerStarted","Data":"53714ce5fecc575a6ec30ca5e7ec6fa04604b0de37b3276eb82ee5fdd36a90fa"} Jul 02 19:07:43.536507 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:43.536418 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dqx74" event={"ID":"e9bceedd-6e94-4988-baf5-5eb9eb417a7b","Type":"ContainerStarted","Data":"537ca8bd64a9ec4c92ef233e955eb71b0bf912b666d48eb6c24e92064c28be44"} Jul 02 19:07:43.546985 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:43.546896 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l249r" event={"ID":"744e479a-55a4-4de6-985b-a3b13ad4145f","Type":"ContainerStarted","Data":"210ad5785d2b546bc071488f7a42c7de7e96a9ca33afca267ab6c0b7189bf1fc"} Jul 02 19:07:43.641567 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:43.641507 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:07:43.913662 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:43.911574 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret\") pod \"global-pull-secret-syncer-jfl78\" (UID: \"f714a87f-b1c3-4e57-8a2f-c1f61ab315be\") " pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:43.920914 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:43.917606 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:07:43.920914 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:43.917709 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret podName:f714a87f-b1c3-4e57-8a2f-c1f61ab315be nodeName:}" failed. No retries permitted until 2026-07-02 19:07:45.917687694 +0000 UTC m=+5.033038914 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret") pod "global-pull-secret-syncer-jfl78" (UID: "f714a87f-b1c3-4e57-8a2f-c1f61ab315be") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:07:44.013264 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:44.013222 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhr4b\" (UniqueName: \"kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b\") pod \"network-check-target-m66fc\" (UID: \"b1b2b08c-8ac4-431a-92df-15eb9b42cc4b\") " pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:07:44.013462 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:44.013320 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs\") pod \"network-metrics-daemon-ld8fm\" (UID: \"e21bc13b-a1b4-4934-9f3e-7eb280915f9b\") " pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:07:44.013528 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:44.013460 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:07:44.013573 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:44.013531 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs podName:e21bc13b-a1b4-4934-9f3e-7eb280915f9b nodeName:}" failed. No retries permitted until 2026-07-02 19:07:46.013510325 +0000 UTC m=+5.128861543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs") pod "network-metrics-daemon-ld8fm" (UID: "e21bc13b-a1b4-4934-9f3e-7eb280915f9b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:07:44.013964 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:44.013940 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:07:44.013964 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:44.013968 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:07:44.014126 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:44.013981 2565 projected.go:196] Error preparing data for projected volume kube-api-access-jhr4b for pod openshift-network-diagnostics/network-check-target-m66fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:07:44.014126 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:44.014029 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b podName:b1b2b08c-8ac4-431a-92df-15eb9b42cc4b nodeName:}" failed. No retries permitted until 2026-07-02 19:07:46.014013934 +0000 UTC m=+5.129365151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jhr4b" (UniqueName: "kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b") pod "network-check-target-m66fc" (UID: "b1b2b08c-8ac4-431a-92df-15eb9b42cc4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:07:44.235823 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:44.235562 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:07:44.337828 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:44.337739 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 19:02:42 +0000 UTC" deadline="2028-01-04 05:17:37.468864873 +0000 UTC" Jul 02 19:07:44.337828 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:44.337803 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13210h9m53.131066231s" Jul 02 19:07:44.394490 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:44.394458 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:07:44.394668 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:44.394601 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ld8fm" podUID="e21bc13b-a1b4-4934-9f3e-7eb280915f9b" Jul 02 19:07:44.395325 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:44.394998 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:07:44.395325 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:44.395090 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m66fc" podUID="b1b2b08c-8ac4-431a-92df-15eb9b42cc4b" Jul 02 19:07:44.395325 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:44.395157 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:44.395325 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:44.395227 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfl78" podUID="f714a87f-b1c3-4e57-8a2f-c1f61ab315be" Jul 02 19:07:45.930712 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:45.930673 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret\") pod \"global-pull-secret-syncer-jfl78\" (UID: \"f714a87f-b1c3-4e57-8a2f-c1f61ab315be\") " pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:45.931249 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:45.930828 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:07:45.931249 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:45.930893 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret podName:f714a87f-b1c3-4e57-8a2f-c1f61ab315be nodeName:}" failed. No retries permitted until 2026-07-02 19:07:49.930874735 +0000 UTC m=+9.046225948 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret") pod "global-pull-secret-syncer-jfl78" (UID: "f714a87f-b1c3-4e57-8a2f-c1f61ab315be") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:07:46.032837 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:46.032023 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhr4b\" (UniqueName: \"kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b\") pod \"network-check-target-m66fc\" (UID: \"b1b2b08c-8ac4-431a-92df-15eb9b42cc4b\") " pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:07:46.032837 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:46.032161 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs\") pod \"network-metrics-daemon-ld8fm\" (UID: \"e21bc13b-a1b4-4934-9f3e-7eb280915f9b\") " pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:07:46.032837 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:46.032269 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:07:46.032837 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:46.032296 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:07:46.032837 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:46.032298 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:07:46.032837 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:46.032310 2565 projected.go:196] Error preparing data for projected volume kube-api-access-jhr4b for pod openshift-network-diagnostics/network-check-target-m66fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:07:46.032837 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:46.032376 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs podName:e21bc13b-a1b4-4934-9f3e-7eb280915f9b nodeName:}" failed. No retries permitted until 2026-07-02 19:07:50.032341397 +0000 UTC m=+9.147692601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs") pod "network-metrics-daemon-ld8fm" (UID: "e21bc13b-a1b4-4934-9f3e-7eb280915f9b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:07:46.032837 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:46.032397 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b podName:b1b2b08c-8ac4-431a-92df-15eb9b42cc4b nodeName:}" failed. No retries permitted until 2026-07-02 19:07:50.032386828 +0000 UTC m=+9.147738033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jhr4b" (UniqueName: "kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b") pod "network-check-target-m66fc" (UID: "b1b2b08c-8ac4-431a-92df-15eb9b42cc4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:07:46.394048 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:46.393973 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:07:46.394202 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:46.394114 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ld8fm" podUID="e21bc13b-a1b4-4934-9f3e-7eb280915f9b" Jul 02 19:07:46.394802 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:46.394521 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:07:46.394802 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:46.394609 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m66fc" podUID="b1b2b08c-8ac4-431a-92df-15eb9b42cc4b" Jul 02 19:07:46.394802 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:46.394663 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:46.394802 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:46.394733 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfl78" podUID="f714a87f-b1c3-4e57-8a2f-c1f61ab315be" Jul 02 19:07:48.394488 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:48.394453 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:07:48.394971 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:48.394581 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m66fc" podUID="b1b2b08c-8ac4-431a-92df-15eb9b42cc4b" Jul 02 19:07:48.394971 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:48.394646 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:48.394971 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:48.394717 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfl78" podUID="f714a87f-b1c3-4e57-8a2f-c1f61ab315be" Jul 02 19:07:48.394971 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:48.394761 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:07:48.394971 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:48.394843 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ld8fm" podUID="e21bc13b-a1b4-4934-9f3e-7eb280915f9b" Jul 02 19:07:49.966610 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:49.966566 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret\") pod \"global-pull-secret-syncer-jfl78\" (UID: \"f714a87f-b1c3-4e57-8a2f-c1f61ab315be\") " pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:49.967060 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:49.966778 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:07:49.967060 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:49.966854 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret podName:f714a87f-b1c3-4e57-8a2f-c1f61ab315be nodeName:}" failed. No retries permitted until 2026-07-02 19:07:57.966834196 +0000 UTC m=+17.082185398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret") pod "global-pull-secret-syncer-jfl78" (UID: "f714a87f-b1c3-4e57-8a2f-c1f61ab315be") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:07:50.067469 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:50.067428 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs\") pod \"network-metrics-daemon-ld8fm\" (UID: \"e21bc13b-a1b4-4934-9f3e-7eb280915f9b\") " pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:07:50.067647 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:50.067505 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhr4b\" (UniqueName: \"kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b\") pod \"network-check-target-m66fc\" (UID: \"b1b2b08c-8ac4-431a-92df-15eb9b42cc4b\") " pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:07:50.067647 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:50.067632 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:07:50.067757 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:50.067700 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:07:50.067757 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:50.067709 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs podName:e21bc13b-a1b4-4934-9f3e-7eb280915f9b nodeName:}" failed. No retries permitted until 2026-07-02 19:07:58.067687516 +0000 UTC m=+17.183038726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs") pod "network-metrics-daemon-ld8fm" (UID: "e21bc13b-a1b4-4934-9f3e-7eb280915f9b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:07:50.067757 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:50.067721 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:07:50.067757 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:50.067735 2565 projected.go:196] Error preparing data for projected volume kube-api-access-jhr4b for pod openshift-network-diagnostics/network-check-target-m66fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:07:50.067917 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:50.067786 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b podName:b1b2b08c-8ac4-431a-92df-15eb9b42cc4b nodeName:}" failed. No retries permitted until 2026-07-02 19:07:58.067769299 +0000 UTC m=+17.183120511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jhr4b" (UniqueName: "kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b") pod "network-check-target-m66fc" (UID: "b1b2b08c-8ac4-431a-92df-15eb9b42cc4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:07:50.393932 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:50.393895 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:07:50.394121 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:50.394048 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ld8fm" podUID="e21bc13b-a1b4-4934-9f3e-7eb280915f9b" Jul 02 19:07:50.394635 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:50.394431 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:07:50.394635 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:50.394537 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m66fc" podUID="b1b2b08c-8ac4-431a-92df-15eb9b42cc4b" Jul 02 19:07:50.394635 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:50.394585 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:50.394871 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:50.394641 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfl78" podUID="f714a87f-b1c3-4e57-8a2f-c1f61ab315be" Jul 02 19:07:52.394692 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:52.394530 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:52.394692 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:52.394558 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:07:52.394692 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:52.394529 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:07:52.394692 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:52.394668 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfl78" podUID="f714a87f-b1c3-4e57-8a2f-c1f61ab315be" Jul 02 19:07:52.395319 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:52.394761 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m66fc" podUID="b1b2b08c-8ac4-431a-92df-15eb9b42cc4b" Jul 02 19:07:52.395319 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:52.394857 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ld8fm" podUID="e21bc13b-a1b4-4934-9f3e-7eb280915f9b" Jul 02 19:07:54.394367 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:54.394320 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:07:54.394778 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:54.394320 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:07:54.394778 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:54.394462 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m66fc" podUID="b1b2b08c-8ac4-431a-92df-15eb9b42cc4b" Jul 02 19:07:54.394778 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:54.394580 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ld8fm" podUID="e21bc13b-a1b4-4934-9f3e-7eb280915f9b" Jul 02 19:07:54.394778 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:54.394320 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:54.394778 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:54.394721 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfl78" podUID="f714a87f-b1c3-4e57-8a2f-c1f61ab315be" Jul 02 19:07:56.394787 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:56.394747 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:07:56.395260 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:56.394877 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:56.395260 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:56.394881 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m66fc" podUID="b1b2b08c-8ac4-431a-92df-15eb9b42cc4b" Jul 02 19:07:56.395260 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:56.394914 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:07:56.395260 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:56.395005 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfl78" podUID="f714a87f-b1c3-4e57-8a2f-c1f61ab315be" Jul 02 19:07:56.395260 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:56.395101 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ld8fm" podUID="e21bc13b-a1b4-4934-9f3e-7eb280915f9b" Jul 02 19:07:58.028945 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:58.028775 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret\") pod \"global-pull-secret-syncer-jfl78\" (UID: \"f714a87f-b1c3-4e57-8a2f-c1f61ab315be\") " pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:58.028945 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:58.028929 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:07:58.029472 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:58.029008 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret podName:f714a87f-b1c3-4e57-8a2f-c1f61ab315be nodeName:}" failed. No retries permitted until 2026-07-02 19:08:14.028987601 +0000 UTC m=+33.144338807 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret") pod "global-pull-secret-syncer-jfl78" (UID: "f714a87f-b1c3-4e57-8a2f-c1f61ab315be") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:07:58.129669 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:58.129635 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhr4b\" (UniqueName: \"kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b\") pod \"network-check-target-m66fc\" (UID: \"b1b2b08c-8ac4-431a-92df-15eb9b42cc4b\") " pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:07:58.129829 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:58.129696 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs\") pod \"network-metrics-daemon-ld8fm\" (UID: \"e21bc13b-a1b4-4934-9f3e-7eb280915f9b\") " pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:07:58.129829 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:58.129801 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:07:58.129956 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:58.129833 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:07:58.129956 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:58.129857 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs podName:e21bc13b-a1b4-4934-9f3e-7eb280915f9b nodeName:}" failed. No retries permitted until 2026-07-02 19:08:14.129841644 +0000 UTC m=+33.245192844 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs") pod "network-metrics-daemon-ld8fm" (UID: "e21bc13b-a1b4-4934-9f3e-7eb280915f9b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:07:58.129956 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:58.129858 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:07:58.129956 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:58.129873 2565 projected.go:196] Error preparing data for projected volume kube-api-access-jhr4b for pod openshift-network-diagnostics/network-check-target-m66fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:07:58.129956 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:58.129925 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b podName:b1b2b08c-8ac4-431a-92df-15eb9b42cc4b nodeName:}" failed. No retries permitted until 2026-07-02 19:08:14.129907713 +0000 UTC m=+33.245258936 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jhr4b" (UniqueName: "kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b") pod "network-check-target-m66fc" (UID: "b1b2b08c-8ac4-431a-92df-15eb9b42cc4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:07:58.394406 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:58.394307 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:07:58.394406 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:58.394337 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:07:58.394638 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:07:58.394328 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:07:58.394638 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:58.394460 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ld8fm" podUID="e21bc13b-a1b4-4934-9f3e-7eb280915f9b" Jul 02 19:07:58.394638 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:58.394541 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m66fc" podUID="b1b2b08c-8ac4-431a-92df-15eb9b42cc4b" Jul 02 19:07:58.394638 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:07:58.394611 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfl78" podUID="f714a87f-b1c3-4e57-8a2f-c1f61ab315be" Jul 02 19:08:00.394473 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:00.394429 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:08:00.394936 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:00.394480 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:08:00.394936 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:00.394485 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:08:00.394936 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:00.394555 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m66fc" podUID="b1b2b08c-8ac4-431a-92df-15eb9b42cc4b" Jul 02 19:08:00.394936 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:00.394695 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ld8fm" podUID="e21bc13b-a1b4-4934-9f3e-7eb280915f9b" Jul 02 19:08:00.394936 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:00.394792 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfl78" podUID="f714a87f-b1c3-4e57-8a2f-c1f61ab315be" Jul 02 19:08:02.394535 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:02.394502 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:08:02.394979 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:02.394599 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m66fc" podUID="b1b2b08c-8ac4-431a-92df-15eb9b42cc4b" Jul 02 19:08:02.394979 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:02.394512 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:08:02.394979 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:02.394651 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:08:02.394979 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:02.394801 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ld8fm" podUID="e21bc13b-a1b4-4934-9f3e-7eb280915f9b" Jul 02 19:08:02.394979 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:02.394893 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfl78" podUID="f714a87f-b1c3-4e57-8a2f-c1f61ab315be" Jul 02 19:08:03.589718 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.589529 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x6shn" event={"ID":"315e9e92-0d3b-48f2-87d5-e3819e54fc0f","Type":"ContainerStarted","Data":"421d7fab63e95f7a500f2ea5d6f1fe57511f07e9a70c2133717ed8b7620b4101"} Jul 02 19:08:03.591128 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.591095 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" event={"ID":"f410247b-ac6f-4294-abf0-f34f1a57db0c","Type":"ContainerStarted","Data":"eab3003f46ef202a66b576b8d45e9c58284e7b4f0d05a341fc79f4c06c0da25f"} Jul 02 19:08:03.592535 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.592513 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mxvgp" event={"ID":"229973e8-3912-4f78-8d39-3f4a49ef2847","Type":"ContainerStarted","Data":"1794adda39b199262ed7c965883d944a7ee7a1ff7923bf46144dd339e0c3ea0b"} Jul 02 19:08:03.594036 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.594012 2565 generic.go:356] "Generic (PLEG): container finished" podID="f74042b81dfcc4a8623a871af2883386" containerID="6c26923a043d31786af294b9c9a529882554978d9de0d8432854ad43fb25e544" exitCode=0 Jul 02 19:08:03.594121 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.594086 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal" event={"ID":"f74042b81dfcc4a8623a871af2883386","Type":"ContainerDied","Data":"6c26923a043d31786af294b9c9a529882554978d9de0d8432854ad43fb25e544"} Jul 02 19:08:03.597014 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.596978 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-165.ec2.internal" event={"ID":"7a9b387f1ea494ba8e3440d0784de7c5","Type":"ContainerStarted","Data":"efadc92c7bc6c040618163208d4ce4e9d4ef3f94763f68fdcb3ac1ff0c247b26"} Jul 02 19:08:03.598871 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.598842 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hf8w8" event={"ID":"b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39","Type":"ContainerStarted","Data":"a6d4e7e9b45d0457c90f73ef2591fcefbcc897ce61ebeb3ee08fb819ad5c4da6"} Jul 02 19:08:03.601391 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.601341 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" event={"ID":"64cd41f8-0247-478f-bc3e-c16135f46632","Type":"ContainerStarted","Data":"7df0a2f17bd0e051f76b7a422f0da3cf90b2c5e4fc8f15638fa27a2f248668b7"} Jul 02 19:08:03.601492 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.601443 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" event={"ID":"64cd41f8-0247-478f-bc3e-c16135f46632","Type":"ContainerStarted","Data":"2080a4477813d05ca34a8eeca61f613a3cb22d18ced77210222c625b611b363d"} Jul 02 19:08:03.601548 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.601497 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" event={"ID":"64cd41f8-0247-478f-bc3e-c16135f46632","Type":"ContainerStarted","Data":"ff65b981d647ae53240cd75df5307a80e225d586e6598d45674f56104128aac2"} Jul 02 19:08:03.601597 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.601547 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" event={"ID":"64cd41f8-0247-478f-bc3e-c16135f46632","Type":"ContainerStarted","Data":"18ad0439b5c93ef17bec06e0dab8b0f7d6eb3ced485290a5a4d46e5d3b31de94"} Jul 02 19:08:03.601650 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.601597 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" event={"ID":"64cd41f8-0247-478f-bc3e-c16135f46632","Type":"ContainerStarted","Data":"93bf7f6f359f6ad8e8f9c5c523d029dbce646530a8ce2dfcf0a5fac897c02a84"} Jul 02 19:08:03.601705 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.601647 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" event={"ID":"64cd41f8-0247-478f-bc3e-c16135f46632","Type":"ContainerStarted","Data":"0ce1d36a6cec707313731cea19c5685c30b0ee62a486e63d6776cc77a93f45f7"} Jul 02 19:08:03.602612 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.602586 2565 generic.go:356] "Generic (PLEG): container finished" podID="ce6ec4d4-1f88-40e4-b020-e1e137efd53a" containerID="ccb7f726fefe2f4b113995c8b7e4a55c89e41ec3d1099c8c84f4063743045556" exitCode=0 Jul 02 19:08:03.602680 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.602658 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgwb4" event={"ID":"ce6ec4d4-1f88-40e4-b020-e1e137efd53a","Type":"ContainerDied","Data":"ccb7f726fefe2f4b113995c8b7e4a55c89e41ec3d1099c8c84f4063743045556"} Jul 02 19:08:03.603940 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.603891 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dqx74" event={"ID":"e9bceedd-6e94-4988-baf5-5eb9eb417a7b","Type":"ContainerStarted","Data":"8a212c62748215610026081eddcb00ba60a98d3e01531e4249373c182b720d5a"} Jul 02 19:08:03.605631 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.605584 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x6shn" podStartSLOduration=2.760032638 podStartE2EDuration="22.605567784s" totalImagesPullingTime="19.845535146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:03.605556978 +0000 UTC m=+22.720908227" watchObservedRunningTime="2026-07-02 19:08:03.605567784 +0000 UTC m=+22.720919008" Jul 02 19:08:03.605891 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.605864 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-l249r" event={"ID":"744e479a-55a4-4de6-985b-a3b13ad4145f","Type":"ContainerStarted","Data":"8c9ea62df04c7549dae69dcb95b24d11c7010fb798765c151b71462d47a1e052"} Jul 02 19:08:03.624916 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.624871 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hf8w8" podStartSLOduration=2.719772486 podStartE2EDuration="22.624855145s" totalImagesPullingTime="19.905082659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:03.624648428 +0000 UTC m=+22.739999651" watchObservedRunningTime="2026-07-02 19:08:03.624855145 +0000 UTC m=+22.740206370" Jul 02 19:08:03.656567 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.656514 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dqx74" podStartSLOduration=2.7085425499999998 podStartE2EDuration="22.656492294s" totalImagesPullingTime="19.947949744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:03.656092176 +0000 UTC m=+22.771443400" watchObservedRunningTime="2026-07-02 19:08:03.656492294 +0000 UTC m=+22.771843518" Jul 02 19:08:03.696085 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.695914 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-165.ec2.internal" podStartSLOduration=21.695897786 podStartE2EDuration="21.695897786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:03.69540149 +0000 UTC m=+22.810752730" watchObservedRunningTime="2026-07-02 19:08:03.695897786 +0000 UTC m=+22.811249008" Jul 02 19:08:03.719944 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.719897 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mxvgp" podStartSLOduration=2.677138734 podStartE2EDuration="22.719876414s" totalImagesPullingTime="20.04273768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:03.719382081 +0000 UTC m=+22.834733328" watchObservedRunningTime="2026-07-02 19:08:03.719876414 +0000 UTC m=+22.835227637" Jul 02 19:08:03.739512 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:03.739450 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-l249r" podStartSLOduration=2.735489845 podStartE2EDuration="22.739431444s" totalImagesPullingTime="20.003941599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:03.738826534 +0000 UTC m=+22.854177757" watchObservedRunningTime="2026-07-02 19:08:03.739431444 +0000 UTC m=+22.854782666" Jul 02 19:08:04.394509 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:04.394428 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:08:04.394509 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:04.394451 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:08:04.394757 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:04.394423 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:08:04.394757 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:04.394559 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m66fc" podUID="b1b2b08c-8ac4-431a-92df-15eb9b42cc4b" Jul 02 19:08:04.394757 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:04.394647 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ld8fm" podUID="e21bc13b-a1b4-4934-9f3e-7eb280915f9b" Jul 02 19:08:04.394757 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:04.394733 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfl78" podUID="f714a87f-b1c3-4e57-8a2f-c1f61ab315be" Jul 02 19:08:04.609632 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:04.609596 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-swfnn" event={"ID":"3b920bdb-1920-4df3-a4d3-cd710aec8492","Type":"ContainerStarted","Data":"615b653c38b548dcaceceac6b6496f7b36b32480362a031bab40c90150817881"} Jul 02 19:08:04.611556 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:04.611524 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal" event={"ID":"f74042b81dfcc4a8623a871af2883386","Type":"ContainerStarted","Data":"d1d60be431258920b403e88dcb6d7ce952b22db09919e8577c3512b54d22202c"} Jul 02 19:08:04.635863 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:04.635812 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-swfnn" podStartSLOduration=3.679877861 podStartE2EDuration="23.635795045s" totalImagesPullingTime="19.955917184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:04.635037897 +0000 UTC m=+23.750389146" watchObservedRunningTime="2026-07-02 19:08:04.635795045 +0000 UTC m=+23.751146268" Jul 02 19:08:04.649341 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:04.649248 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-165.ec2.internal" podStartSLOduration=22.649227839 podStartE2EDuration="22.649227839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:04.648511955 +0000 UTC m=+23.763863202" watchObservedRunningTime="2026-07-02 19:08:04.649227839 +0000 UTC m=+23.764579064" Jul 02 19:08:04.749944 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:04.749914 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 02 19:08:05.387087 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:05.386946 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T19:08:04.749934626Z","UUID":"2ed3f48a-b516-402b-bf3e-b26e170b35df","Handler":null,"Name":"","Endpoint":""} Jul 02 19:08:05.388673 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:05.388641 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 02 19:08:05.388800 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:05.388693 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 02 19:08:05.615800 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:05.615756 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" event={"ID":"f410247b-ac6f-4294-abf0-f34f1a57db0c","Type":"ContainerStarted","Data":"c273d0d9f7fc0076d29efa09f57e6fa9ff981e8a2e85e9f6ce1008208afba4dd"} Jul 02 19:08:06.394575 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:06.394494 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:08:06.394575 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:06.394543 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:08:06.394575 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:06.394544 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:08:06.394944 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:06.394625 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfl78" podUID="f714a87f-b1c3-4e57-8a2f-c1f61ab315be" Jul 02 19:08:06.394944 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:06.394723 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m66fc" podUID="b1b2b08c-8ac4-431a-92df-15eb9b42cc4b" Jul 02 19:08:06.394944 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:06.394827 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ld8fm" podUID="e21bc13b-a1b4-4934-9f3e-7eb280915f9b" Jul 02 19:08:06.621019 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:06.620981 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" event={"ID":"f410247b-ac6f-4294-abf0-f34f1a57db0c","Type":"ContainerStarted","Data":"73c816d83985943404f14bc03c736c86d7222b443db681cc7653b9ab3c3e359f"} Jul 02 19:08:06.628192 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:06.628157 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" event={"ID":"64cd41f8-0247-478f-bc3e-c16135f46632","Type":"ContainerStarted","Data":"094ff6f600c0b0a73b9734a9f23dba3df5e42711a23e5d6446c7e598f65c7d76"} Jul 02 19:08:06.638802 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:06.638748 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6qq7l" podStartSLOduration=3.281868372 podStartE2EDuration="25.63872929s" totalImagesPullingTime="22.356860918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:07:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:06.638431155 +0000 UTC m=+25.753782404" watchObservedRunningTime="2026-07-02 19:08:06.63872929 +0000 UTC m=+25.754080513" Jul 02 19:08:07.393222 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:07.393185 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x6shn" Jul 02 19:08:07.396780 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:07.396758 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x6shn" Jul 02 19:08:07.630452 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:07.630426 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x6shn" Jul 02 19:08:07.630905 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:07.630802 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x6shn" Jul 02 19:08:08.394543 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:08.394501 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:08:08.394723 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:08.394501 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:08:08.394723 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:08.394501 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:08:08.394723 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:08.394693 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfl78" podUID="f714a87f-b1c3-4e57-8a2f-c1f61ab315be" Jul 02 19:08:08.394817 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:08.394612 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m66fc" podUID="b1b2b08c-8ac4-431a-92df-15eb9b42cc4b" Jul 02 19:08:08.394817 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:08.394764 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ld8fm" podUID="e21bc13b-a1b4-4934-9f3e-7eb280915f9b" Jul 02 19:08:08.635583 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:08.635548 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" event={"ID":"64cd41f8-0247-478f-bc3e-c16135f46632","Type":"ContainerStarted","Data":"66e91cca593dc3bd0abae7c0043263edd8bd315b8377bc9a0c8e39e7b5c787ba"} Jul 02 19:08:08.636217 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:08.635798 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:08:08.636217 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:08.635873 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:08:08.636217 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:08.635918 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:08:08.637670 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:08.637641 2565 generic.go:356] "Generic (PLEG): container finished" podID="ce6ec4d4-1f88-40e4-b020-e1e137efd53a" containerID="0b273802044186ff94ac660c4b15246aeb0d9ad7d942092df7ddc025e48ae89b" exitCode=0 Jul 02 19:08:08.637806 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:08.637700 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgwb4" event={"ID":"ce6ec4d4-1f88-40e4-b020-e1e137efd53a","Type":"ContainerDied","Data":"0b273802044186ff94ac660c4b15246aeb0d9ad7d942092df7ddc025e48ae89b"} Jul 02 19:08:08.651726 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:08.651654 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:08:08.651889 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:08.651872 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:08:08.661302 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:08.661257 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" podStartSLOduration=7.570018099 podStartE2EDuration="27.66124405s" totalImagesPullingTime="20.091225951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:08.660809074 +0000 UTC m=+27.776160298" watchObservedRunningTime="2026-07-02 19:08:08.66124405 +0000 UTC m=+27.776595273" Jul 02 19:08:09.792785 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:09.792598 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jfl78"] Jul 02 19:08:09.793300 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:09.792873 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:08:09.793300 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:09.792958 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfl78" podUID="f714a87f-b1c3-4e57-8a2f-c1f61ab315be" Jul 02 19:08:09.795250 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:09.795220 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m66fc"] Jul 02 19:08:09.795391 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:09.795321 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:08:09.795470 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:09.795441 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m66fc" podUID="b1b2b08c-8ac4-431a-92df-15eb9b42cc4b" Jul 02 19:08:09.809866 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:09.809843 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ld8fm"] Jul 02 19:08:09.809980 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:09.809945 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:08:09.810036 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:09.810021 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ld8fm" podUID="e21bc13b-a1b4-4934-9f3e-7eb280915f9b" Jul 02 19:08:10.646004 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:10.645934 2565 generic.go:356] "Generic (PLEG): container finished" podID="ce6ec4d4-1f88-40e4-b020-e1e137efd53a" containerID="5921b9580061d3492dc24875796b841a89028898d6e1da3c2d909110b6a165df" exitCode=0 Jul 02 19:08:10.646184 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:10.646014 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgwb4" event={"ID":"ce6ec4d4-1f88-40e4-b020-e1e137efd53a","Type":"ContainerDied","Data":"5921b9580061d3492dc24875796b841a89028898d6e1da3c2d909110b6a165df"} Jul 02 19:08:11.396849 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:11.396819 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:08:11.397301 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:11.396909 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:08:11.397301 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:11.396915 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfl78" podUID="f714a87f-b1c3-4e57-8a2f-c1f61ab315be" Jul 02 19:08:11.397301 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:11.397038 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ld8fm" podUID="e21bc13b-a1b4-4934-9f3e-7eb280915f9b" Jul 02 19:08:11.397301 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:11.397096 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:08:11.397301 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:11.397171 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m66fc" podUID="b1b2b08c-8ac4-431a-92df-15eb9b42cc4b" Jul 02 19:08:11.651874 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:11.651830 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgwb4" event={"ID":"ce6ec4d4-1f88-40e4-b020-e1e137efd53a","Type":"ContainerStarted","Data":"49a7c5fbbce7f4c6a2265faffe97058d6c09d86308944451a24a571dcf8fd5c2"} Jul 02 19:08:12.656223 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:12.656193 2565 generic.go:356] "Generic (PLEG): container finished" podID="ce6ec4d4-1f88-40e4-b020-e1e137efd53a" containerID="49a7c5fbbce7f4c6a2265faffe97058d6c09d86308944451a24a571dcf8fd5c2" exitCode=0 Jul 02 19:08:12.656704 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:12.656251 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgwb4" event={"ID":"ce6ec4d4-1f88-40e4-b020-e1e137efd53a","Type":"ContainerDied","Data":"49a7c5fbbce7f4c6a2265faffe97058d6c09d86308944451a24a571dcf8fd5c2"} Jul 02 19:08:13.394835 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:13.394801 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:08:13.394835 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:13.394833 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:08:13.395081 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:13.394801 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:08:13.395081 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:13.394952 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ld8fm" podUID="e21bc13b-a1b4-4934-9f3e-7eb280915f9b" Jul 02 19:08:13.395180 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:13.395079 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfl78" podUID="f714a87f-b1c3-4e57-8a2f-c1f61ab315be" Jul 02 19:08:13.395212 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:13.395178 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m66fc" podUID="b1b2b08c-8ac4-431a-92df-15eb9b42cc4b" Jul 02 19:08:14.044407 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:14.044361 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret\") pod \"global-pull-secret-syncer-jfl78\" (UID: \"f714a87f-b1c3-4e57-8a2f-c1f61ab315be\") " pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:08:14.044814 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:14.044483 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:08:14.044814 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:14.044571 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret podName:f714a87f-b1c3-4e57-8a2f-c1f61ab315be nodeName:}" failed. No retries permitted until 2026-07-02 19:08:46.044548815 +0000 UTC m=+65.159900020 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret") pod "global-pull-secret-syncer-jfl78" (UID: "f714a87f-b1c3-4e57-8a2f-c1f61ab315be") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:08:14.145503 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:14.145462 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs\") pod \"network-metrics-daemon-ld8fm\" (UID: \"e21bc13b-a1b4-4934-9f3e-7eb280915f9b\") " pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:08:14.145698 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:14.145529 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhr4b\" (UniqueName: \"kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b\") pod \"network-check-target-m66fc\" (UID: \"b1b2b08c-8ac4-431a-92df-15eb9b42cc4b\") " pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:08:14.145698 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:14.145657 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:08:14.145698 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:14.145685 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:08:14.145855 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:14.145702 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:08:14.145855 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:14.145715 2565 projected.go:196] Error preparing data for projected volume kube-api-access-jhr4b for pod openshift-network-diagnostics/network-check-target-m66fc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:08:14.145855 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:14.145775 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs podName:e21bc13b-a1b4-4934-9f3e-7eb280915f9b nodeName:}" failed. No retries permitted until 2026-07-02 19:08:46.145714329 +0000 UTC m=+65.261065531 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs") pod "network-metrics-daemon-ld8fm" (UID: "e21bc13b-a1b4-4934-9f3e-7eb280915f9b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:08:14.145855 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:14.145798 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b podName:b1b2b08c-8ac4-431a-92df-15eb9b42cc4b nodeName:}" failed. No retries permitted until 2026-07-02 19:08:46.14578726 +0000 UTC m=+65.261138468 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jhr4b" (UniqueName: "kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b") pod "network-check-target-m66fc" (UID: "b1b2b08c-8ac4-431a-92df-15eb9b42cc4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:08:15.394498 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:15.394460 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:08:15.394918 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:15.394460 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:08:15.394918 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:15.394601 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfl78" podUID="f714a87f-b1c3-4e57-8a2f-c1f61ab315be" Jul 02 19:08:15.394918 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:15.394613 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:08:15.394918 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:15.394768 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ld8fm" podUID="e21bc13b-a1b4-4934-9f3e-7eb280915f9b" Jul 02 19:08:15.394918 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:15.394865 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m66fc" podUID="b1b2b08c-8ac4-431a-92df-15eb9b42cc4b" Jul 02 19:08:16.669314 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.669146 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgwb4" event={"ID":"ce6ec4d4-1f88-40e4-b020-e1e137efd53a","Type":"ContainerStarted","Data":"a537cf15149f36dac82b05e1d6531cc5f98f14e38770178eae68cb2e8fbb5ca2"} Jul 02 19:08:16.701903 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.701875 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-165.ec2.internal" event="NodeReady" Jul 02 19:08:16.702061 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.702030 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 02 19:08:16.757255 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.757225 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8fcgj"] Jul 02 19:08:16.759562 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.759546 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:16.761935 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.761909 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wlnv6"] Jul 02 19:08:16.765630 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.762687 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 02 19:08:16.765630 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.763278 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-45sjn\"" Jul 02 19:08:16.765630 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.763373 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 02 19:08:16.766094 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.766078 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wlnv6" Jul 02 19:08:16.768864 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.768844 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 02 19:08:16.768864 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.768864 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 02 19:08:16.769040 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.768852 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 02 19:08:16.771458 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.771440 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pjv9d\"" Jul 02 19:08:16.778218 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.778199 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wlnv6"] Jul 02 19:08:16.782200 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.782180 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8fcgj"] Jul 02 19:08:16.866943 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.866847 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-config-volume\") pod \"dns-default-8fcgj\" (UID: \"442ba05b-cbb8-48fb-b1ab-96ebe4960f80\") " pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:16.866943 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.866889 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qzj5\" (UniqueName: \"kubernetes.io/projected/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-kube-api-access-4qzj5\") pod \"dns-default-8fcgj\" (UID: \"442ba05b-cbb8-48fb-b1ab-96ebe4960f80\") " pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:16.866943 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.866922 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-tmp-dir\") pod \"dns-default-8fcgj\" (UID: \"442ba05b-cbb8-48fb-b1ab-96ebe4960f80\") " pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:16.866943 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.866942 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e81d42fb-8cb6-4e07-863d-4912aa0ed5e8-cert\") pod \"ingress-canary-wlnv6\" (UID: \"e81d42fb-8cb6-4e07-863d-4912aa0ed5e8\") " pod="openshift-ingress-canary/ingress-canary-wlnv6" Jul 02 19:08:16.867216 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.866969 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls\") pod \"dns-default-8fcgj\" (UID: \"442ba05b-cbb8-48fb-b1ab-96ebe4960f80\") " pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:16.867216 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.867034 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lnwf\" (UniqueName: \"kubernetes.io/projected/e81d42fb-8cb6-4e07-863d-4912aa0ed5e8-kube-api-access-6lnwf\") pod \"ingress-canary-wlnv6\" (UID: \"e81d42fb-8cb6-4e07-863d-4912aa0ed5e8\") " pod="openshift-ingress-canary/ingress-canary-wlnv6" Jul 02 19:08:16.967702 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.967664 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lnwf\" (UniqueName: \"kubernetes.io/projected/e81d42fb-8cb6-4e07-863d-4912aa0ed5e8-kube-api-access-6lnwf\") pod \"ingress-canary-wlnv6\" (UID: \"e81d42fb-8cb6-4e07-863d-4912aa0ed5e8\") " pod="openshift-ingress-canary/ingress-canary-wlnv6" Jul 02 19:08:16.967895 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.967756 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-config-volume\") pod \"dns-default-8fcgj\" (UID: \"442ba05b-cbb8-48fb-b1ab-96ebe4960f80\") " pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:16.967895 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.967776 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qzj5\" (UniqueName: \"kubernetes.io/projected/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-kube-api-access-4qzj5\") pod \"dns-default-8fcgj\" (UID: \"442ba05b-cbb8-48fb-b1ab-96ebe4960f80\") " pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:16.967895 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.967839 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-tmp-dir\") pod \"dns-default-8fcgj\" (UID: \"442ba05b-cbb8-48fb-b1ab-96ebe4960f80\") " pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:16.967895 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.967894 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e81d42fb-8cb6-4e07-863d-4912aa0ed5e8-cert\") pod \"ingress-canary-wlnv6\" (UID: \"e81d42fb-8cb6-4e07-863d-4912aa0ed5e8\") " pod="openshift-ingress-canary/ingress-canary-wlnv6" Jul 02 19:08:16.968107 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.967917 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls\") pod \"dns-default-8fcgj\" (UID: \"442ba05b-cbb8-48fb-b1ab-96ebe4960f80\") " pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:16.968107 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:16.968026 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:08:16.968107 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.968077 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-tmp-dir\") pod \"dns-default-8fcgj\" (UID: \"442ba05b-cbb8-48fb-b1ab-96ebe4960f80\") " pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:16.968107 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:16.968082 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls podName:442ba05b-cbb8-48fb-b1ab-96ebe4960f80 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:17.468064079 +0000 UTC m=+36.583415283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls") pod "dns-default-8fcgj" (UID: "442ba05b-cbb8-48fb-b1ab-96ebe4960f80") : secret "dns-default-metrics-tls" not found Jul 02 19:08:16.968313 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.968295 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-config-volume\") pod \"dns-default-8fcgj\" (UID: \"442ba05b-cbb8-48fb-b1ab-96ebe4960f80\") " pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:16.972152 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.972135 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e81d42fb-8cb6-4e07-863d-4912aa0ed5e8-cert\") pod \"ingress-canary-wlnv6\" (UID: \"e81d42fb-8cb6-4e07-863d-4912aa0ed5e8\") " pod="openshift-ingress-canary/ingress-canary-wlnv6" Jul 02 19:08:16.976701 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.976675 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qzj5\" (UniqueName: \"kubernetes.io/projected/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-kube-api-access-4qzj5\") pod \"dns-default-8fcgj\" (UID: \"442ba05b-cbb8-48fb-b1ab-96ebe4960f80\") " pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:16.977044 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:16.977027 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lnwf\" (UniqueName: \"kubernetes.io/projected/e81d42fb-8cb6-4e07-863d-4912aa0ed5e8-kube-api-access-6lnwf\") pod \"ingress-canary-wlnv6\" (UID: \"e81d42fb-8cb6-4e07-863d-4912aa0ed5e8\") " pod="openshift-ingress-canary/ingress-canary-wlnv6" Jul 02 19:08:17.078411 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.078370 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wlnv6" Jul 02 19:08:17.136093 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.136060 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gqj2t"] Jul 02 19:08:17.139245 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.139220 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gqj2t" Jul 02 19:08:17.141883 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.141859 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 02 19:08:17.141883 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.141872 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:08:17.142015 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.141895 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-rmf45\"" Jul 02 19:08:17.153464 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.153230 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gqj2t"] Jul 02 19:08:17.217327 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.217298 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wlnv6"] Jul 02 19:08:17.221603 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:08:17.221581 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81d42fb_8cb6_4e07_863d_4912aa0ed5e8.slice/crio-080c724cf4d03631ed387f80f08c26d835b832bc56089a3dc3b379028b945e8f WatchSource:0}: Error finding container 080c724cf4d03631ed387f80f08c26d835b832bc56089a3dc3b379028b945e8f: Status 404 returned error can't find the container with id 080c724cf4d03631ed387f80f08c26d835b832bc56089a3dc3b379028b945e8f Jul 02 19:08:17.235706 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.235683 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w"] Jul 02 19:08:17.237854 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.237838 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w" Jul 02 19:08:17.240007 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.239986 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 02 19:08:17.240188 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.240175 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:08:17.240246 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.240196 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 02 19:08:17.240295 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.240276 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jxs6l\"" Jul 02 19:08:17.248171 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.248149 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w"] Jul 02 19:08:17.270776 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.270750 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smdj5\" (UniqueName: \"kubernetes.io/projected/54667ba4-1e72-488c-9fe3-53026b8d2df9-kube-api-access-smdj5\") pod \"volume-data-source-validator-5d456fdbcd-gqj2t\" (UID: \"54667ba4-1e72-488c-9fe3-53026b8d2df9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gqj2t" Jul 02 19:08:17.371192 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.371156 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smdj5\" (UniqueName: \"kubernetes.io/projected/54667ba4-1e72-488c-9fe3-53026b8d2df9-kube-api-access-smdj5\") pod \"volume-data-source-validator-5d456fdbcd-gqj2t\" (UID: \"54667ba4-1e72-488c-9fe3-53026b8d2df9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gqj2t" Jul 02 19:08:17.371311 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.371197 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tz5z\" (UniqueName: \"kubernetes.io/projected/5d408f7b-881f-43f1-89a1-61bcd8cb5261-kube-api-access-9tz5z\") pod \"cluster-samples-operator-56fcf86bf-6cr9w\" (UID: \"5d408f7b-881f-43f1-89a1-61bcd8cb5261\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w" Jul 02 19:08:17.371311 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.371252 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d408f7b-881f-43f1-89a1-61bcd8cb5261-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-6cr9w\" (UID: \"5d408f7b-881f-43f1-89a1-61bcd8cb5261\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w" Jul 02 19:08:17.379240 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.379220 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smdj5\" (UniqueName: \"kubernetes.io/projected/54667ba4-1e72-488c-9fe3-53026b8d2df9-kube-api-access-smdj5\") pod \"volume-data-source-validator-5d456fdbcd-gqj2t\" (UID: \"54667ba4-1e72-488c-9fe3-53026b8d2df9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gqj2t" Jul 02 19:08:17.394397 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.394314 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:08:17.394397 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.394334 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:08:17.394564 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.394546 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:08:17.397070 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.397047 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 19:08:17.397197 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.397156 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 19:08:17.397197 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.397174 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 19:08:17.397197 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.397184 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r7xgd\"" Jul 02 19:08:17.397397 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.397211 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5rf5h\"" Jul 02 19:08:17.397535 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.397522 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 19:08:17.450017 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.449975 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gqj2t" Jul 02 19:08:17.472384 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.472173 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d408f7b-881f-43f1-89a1-61bcd8cb5261-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-6cr9w\" (UID: \"5d408f7b-881f-43f1-89a1-61bcd8cb5261\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w" Jul 02 19:08:17.472536 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.472445 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tz5z\" (UniqueName: \"kubernetes.io/projected/5d408f7b-881f-43f1-89a1-61bcd8cb5261-kube-api-access-9tz5z\") pod \"cluster-samples-operator-56fcf86bf-6cr9w\" (UID: \"5d408f7b-881f-43f1-89a1-61bcd8cb5261\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w" Jul 02 19:08:17.472536 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.472492 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls\") pod \"dns-default-8fcgj\" (UID: \"442ba05b-cbb8-48fb-b1ab-96ebe4960f80\") " pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:17.472536 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:17.472323 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 19:08:17.473078 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:17.473050 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:08:17.473078 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:17.473073 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d408f7b-881f-43f1-89a1-61bcd8cb5261-samples-operator-tls podName:5d408f7b-881f-43f1-89a1-61bcd8cb5261 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:17.973045184 +0000 UTC m=+37.088396399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5d408f7b-881f-43f1-89a1-61bcd8cb5261-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-6cr9w" (UID: "5d408f7b-881f-43f1-89a1-61bcd8cb5261") : secret "samples-operator-tls" not found Jul 02 19:08:17.473241 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:17.473108 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls podName:442ba05b-cbb8-48fb-b1ab-96ebe4960f80 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:18.473096371 +0000 UTC m=+37.588447577 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls") pod "dns-default-8fcgj" (UID: "442ba05b-cbb8-48fb-b1ab-96ebe4960f80") : secret "dns-default-metrics-tls" not found Jul 02 19:08:17.484264 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.484230 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tz5z\" (UniqueName: \"kubernetes.io/projected/5d408f7b-881f-43f1-89a1-61bcd8cb5261-kube-api-access-9tz5z\") pod \"cluster-samples-operator-56fcf86bf-6cr9w\" (UID: \"5d408f7b-881f-43f1-89a1-61bcd8cb5261\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w" Jul 02 19:08:17.576088 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.576051 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gqj2t"] Jul 02 19:08:17.580931 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:08:17.580907 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54667ba4_1e72_488c_9fe3_53026b8d2df9.slice/crio-84f17899eccedff872b69852a03424499cb83f98c9b0af63161406701e29ded5 WatchSource:0}: Error finding container 84f17899eccedff872b69852a03424499cb83f98c9b0af63161406701e29ded5: Status 404 returned error can't find the container with id 84f17899eccedff872b69852a03424499cb83f98c9b0af63161406701e29ded5 Jul 02 19:08:17.673708 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.673674 2565 generic.go:356] "Generic (PLEG): container finished" podID="ce6ec4d4-1f88-40e4-b020-e1e137efd53a" containerID="a537cf15149f36dac82b05e1d6531cc5f98f14e38770178eae68cb2e8fbb5ca2" exitCode=0 Jul 02 19:08:17.674524 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.673754 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgwb4" event={"ID":"ce6ec4d4-1f88-40e4-b020-e1e137efd53a","Type":"ContainerDied","Data":"a537cf15149f36dac82b05e1d6531cc5f98f14e38770178eae68cb2e8fbb5ca2"} Jul 02 19:08:17.674963 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.674938 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wlnv6" event={"ID":"e81d42fb-8cb6-4e07-863d-4912aa0ed5e8","Type":"ContainerStarted","Data":"080c724cf4d03631ed387f80f08c26d835b832bc56089a3dc3b379028b945e8f"} Jul 02 19:08:17.675976 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.675957 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gqj2t" event={"ID":"54667ba4-1e72-488c-9fe3-53026b8d2df9","Type":"ContainerStarted","Data":"84f17899eccedff872b69852a03424499cb83f98c9b0af63161406701e29ded5"} Jul 02 19:08:17.976272 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:17.976167 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d408f7b-881f-43f1-89a1-61bcd8cb5261-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-6cr9w\" (UID: \"5d408f7b-881f-43f1-89a1-61bcd8cb5261\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w" Jul 02 19:08:17.976454 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:17.976413 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 19:08:17.976503 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:17.976483 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d408f7b-881f-43f1-89a1-61bcd8cb5261-samples-operator-tls podName:5d408f7b-881f-43f1-89a1-61bcd8cb5261 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:18.976464678 +0000 UTC m=+38.091815885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5d408f7b-881f-43f1-89a1-61bcd8cb5261-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-6cr9w" (UID: "5d408f7b-881f-43f1-89a1-61bcd8cb5261") : secret "samples-operator-tls" not found Jul 02 19:08:18.120618 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.120583 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-gnf78"] Jul 02 19:08:18.123044 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.123019 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-gnf78" Jul 02 19:08:18.125627 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.125604 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 02 19:08:18.125919 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.125899 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 02 19:08:18.126056 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.126037 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 02 19:08:18.126179 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.126162 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:08:18.126593 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.126567 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-hst5b\"" Jul 02 19:08:18.134285 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.134257 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-gnf78"] Jul 02 19:08:18.242303 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.242213 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xlvvc"] Jul 02 19:08:18.245178 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.245155 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-5pcbl"] Jul 02 19:08:18.245329 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.245279 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xlvvc" Jul 02 19:08:18.247745 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.247294 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" Jul 02 19:08:18.248327 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.248309 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 02 19:08:18.248441 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.248403 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 02 19:08:18.248496 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.248435 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:08:18.248556 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.248306 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 02 19:08:18.248609 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.248552 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-qqvg8\"" Jul 02 19:08:18.249972 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.249920 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 02 19:08:18.250053 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.250025 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-x5m2v\"" Jul 02 19:08:18.250207 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.250190 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 02 19:08:18.250426 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.250411 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:08:18.250510 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.250435 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 02 19:08:18.257426 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.257401 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xlvvc"] Jul 02 19:08:18.261500 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.261465 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-5pcbl"] Jul 02 19:08:18.264792 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.261919 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 02 19:08:18.277949 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.277918 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da2642bb-e400-40ec-bded-5f145bbba018-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-gnf78\" (UID: \"da2642bb-e400-40ec-bded-5f145bbba018\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-gnf78" Jul 02 19:08:18.278092 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.277965 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da2642bb-e400-40ec-bded-5f145bbba018-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-gnf78\" (UID: \"da2642bb-e400-40ec-bded-5f145bbba018\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-gnf78" Jul 02 19:08:18.278169 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.278128 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjchs\" (UniqueName: \"kubernetes.io/projected/da2642bb-e400-40ec-bded-5f145bbba018-kube-api-access-zjchs\") pod \"kube-storage-version-migrator-operator-846bf668f7-gnf78\" (UID: \"da2642bb-e400-40ec-bded-5f145bbba018\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-gnf78" Jul 02 19:08:18.378746 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.378706 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcb6k\" (UniqueName: \"kubernetes.io/projected/fa48a740-00cf-4f9c-815b-c96d531852de-kube-api-access-pcb6k\") pod \"service-ca-operator-5d46d9ffd4-xlvvc\" (UID: \"fa48a740-00cf-4f9c-815b-c96d531852de\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xlvvc" Jul 02 19:08:18.378929 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.378780 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjchs\" (UniqueName: \"kubernetes.io/projected/da2642bb-e400-40ec-bded-5f145bbba018-kube-api-access-zjchs\") pod \"kube-storage-version-migrator-operator-846bf668f7-gnf78\" (UID: \"da2642bb-e400-40ec-bded-5f145bbba018\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-gnf78" Jul 02 19:08:18.378929 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.378809 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fa48a740-00cf-4f9c-815b-c96d531852de-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-xlvvc\" (UID: \"fa48a740-00cf-4f9c-815b-c96d531852de\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xlvvc" Jul 02 19:08:18.378929 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.378849 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fa48a740-00cf-4f9c-815b-c96d531852de-config\") pod \"service-ca-operator-5d46d9ffd4-xlvvc\" (UID: \"fa48a740-00cf-4f9c-815b-c96d531852de\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xlvvc" Jul 02 19:08:18.379084 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.379002 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/39433d94-da63-4417-8cac-bfe000cc31dd-config\") pod \"console-operator-68b757fb67-5pcbl\" (UID: \"39433d94-da63-4417-8cac-bfe000cc31dd\") " pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" Jul 02 19:08:18.379084 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.379056 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da2642bb-e400-40ec-bded-5f145bbba018-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-gnf78\" (UID: \"da2642bb-e400-40ec-bded-5f145bbba018\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-gnf78" Jul 02 19:08:18.379166 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.379090 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2sl6\" (UniqueName: \"kubernetes.io/projected/39433d94-da63-4417-8cac-bfe000cc31dd-kube-api-access-q2sl6\") pod \"console-operator-68b757fb67-5pcbl\" (UID: \"39433d94-da63-4417-8cac-bfe000cc31dd\") " pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" Jul 02 19:08:18.379166 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.379125 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da2642bb-e400-40ec-bded-5f145bbba018-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-gnf78\" (UID: \"da2642bb-e400-40ec-bded-5f145bbba018\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-gnf78" Jul 02 19:08:18.379166 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.379160 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39433d94-da63-4417-8cac-bfe000cc31dd-trusted-ca\") pod \"console-operator-68b757fb67-5pcbl\" (UID: \"39433d94-da63-4417-8cac-bfe000cc31dd\") " pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" Jul 02 19:08:18.379298 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.379250 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/39433d94-da63-4417-8cac-bfe000cc31dd-serving-cert\") pod \"console-operator-68b757fb67-5pcbl\" (UID: \"39433d94-da63-4417-8cac-bfe000cc31dd\") " pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" Jul 02 19:08:18.379753 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.379723 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da2642bb-e400-40ec-bded-5f145bbba018-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-gnf78\" (UID: \"da2642bb-e400-40ec-bded-5f145bbba018\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-gnf78" Jul 02 19:08:18.382483 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.382457 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da2642bb-e400-40ec-bded-5f145bbba018-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-gnf78\" (UID: \"da2642bb-e400-40ec-bded-5f145bbba018\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-gnf78" Jul 02 19:08:18.388401 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.388374 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjchs\" (UniqueName: \"kubernetes.io/projected/da2642bb-e400-40ec-bded-5f145bbba018-kube-api-access-zjchs\") pod \"kube-storage-version-migrator-operator-846bf668f7-gnf78\" (UID: \"da2642bb-e400-40ec-bded-5f145bbba018\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-gnf78" Jul 02 19:08:18.434469 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.434430 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-gnf78" Jul 02 19:08:18.480047 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.480004 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2sl6\" (UniqueName: \"kubernetes.io/projected/39433d94-da63-4417-8cac-bfe000cc31dd-kube-api-access-q2sl6\") pod \"console-operator-68b757fb67-5pcbl\" (UID: \"39433d94-da63-4417-8cac-bfe000cc31dd\") " pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" Jul 02 19:08:18.480311 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.480283 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39433d94-da63-4417-8cac-bfe000cc31dd-trusted-ca\") pod \"console-operator-68b757fb67-5pcbl\" (UID: \"39433d94-da63-4417-8cac-bfe000cc31dd\") " pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" Jul 02 19:08:18.480428 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.480401 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/39433d94-da63-4417-8cac-bfe000cc31dd-serving-cert\") pod \"console-operator-68b757fb67-5pcbl\" (UID: \"39433d94-da63-4417-8cac-bfe000cc31dd\") " pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" Jul 02 19:08:18.480522 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.480493 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcb6k\" (UniqueName: \"kubernetes.io/projected/fa48a740-00cf-4f9c-815b-c96d531852de-kube-api-access-pcb6k\") pod \"service-ca-operator-5d46d9ffd4-xlvvc\" (UID: \"fa48a740-00cf-4f9c-815b-c96d531852de\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xlvvc" Jul 02 19:08:18.480587 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.480555 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls\") pod \"dns-default-8fcgj\" (UID: \"442ba05b-cbb8-48fb-b1ab-96ebe4960f80\") " pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:18.480642 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.480587 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fa48a740-00cf-4f9c-815b-c96d531852de-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-xlvvc\" (UID: \"fa48a740-00cf-4f9c-815b-c96d531852de\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xlvvc" Jul 02 19:08:18.480692 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.480651 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fa48a740-00cf-4f9c-815b-c96d531852de-config\") pod \"service-ca-operator-5d46d9ffd4-xlvvc\" (UID: \"fa48a740-00cf-4f9c-815b-c96d531852de\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xlvvc" Jul 02 19:08:18.480692 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:18.480666 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:08:18.480799 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:18.480732 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls podName:442ba05b-cbb8-48fb-b1ab-96ebe4960f80 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:20.480711589 +0000 UTC m=+39.596062802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls") pod "dns-default-8fcgj" (UID: "442ba05b-cbb8-48fb-b1ab-96ebe4960f80") : secret "dns-default-metrics-tls" not found Jul 02 19:08:18.481034 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.480675 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/39433d94-da63-4417-8cac-bfe000cc31dd-config\") pod \"console-operator-68b757fb67-5pcbl\" (UID: \"39433d94-da63-4417-8cac-bfe000cc31dd\") " pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" Jul 02 19:08:18.481371 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.481330 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fa48a740-00cf-4f9c-815b-c96d531852de-config\") pod \"service-ca-operator-5d46d9ffd4-xlvvc\" (UID: \"fa48a740-00cf-4f9c-815b-c96d531852de\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xlvvc" Jul 02 19:08:18.481496 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.481441 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/39433d94-da63-4417-8cac-bfe000cc31dd-config\") pod \"console-operator-68b757fb67-5pcbl\" (UID: \"39433d94-da63-4417-8cac-bfe000cc31dd\") " pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" Jul 02 19:08:18.482143 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.482121 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/39433d94-da63-4417-8cac-bfe000cc31dd-trusted-ca\") pod \"console-operator-68b757fb67-5pcbl\" (UID: \"39433d94-da63-4417-8cac-bfe000cc31dd\") " pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" Jul 02 19:08:18.483952 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.483926 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fa48a740-00cf-4f9c-815b-c96d531852de-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-xlvvc\" (UID: \"fa48a740-00cf-4f9c-815b-c96d531852de\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xlvvc" Jul 02 19:08:18.484644 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.484606 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/39433d94-da63-4417-8cac-bfe000cc31dd-serving-cert\") pod \"console-operator-68b757fb67-5pcbl\" (UID: \"39433d94-da63-4417-8cac-bfe000cc31dd\") " pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" Jul 02 19:08:18.490569 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.490541 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcb6k\" (UniqueName: \"kubernetes.io/projected/fa48a740-00cf-4f9c-815b-c96d531852de-kube-api-access-pcb6k\") pod \"service-ca-operator-5d46d9ffd4-xlvvc\" (UID: \"fa48a740-00cf-4f9c-815b-c96d531852de\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xlvvc" Jul 02 19:08:18.490885 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.490846 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2sl6\" (UniqueName: \"kubernetes.io/projected/39433d94-da63-4417-8cac-bfe000cc31dd-kube-api-access-q2sl6\") pod \"console-operator-68b757fb67-5pcbl\" (UID: \"39433d94-da63-4417-8cac-bfe000cc31dd\") " pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" Jul 02 19:08:18.562798 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.562760 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xlvvc" Jul 02 19:08:18.571265 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.571039 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" Jul 02 19:08:18.609382 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.608990 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-gnf78"] Jul 02 19:08:18.614299 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:08:18.614197 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2642bb_e400_40ec_bded_5f145bbba018.slice/crio-290d622a95caa3252d069d6b963e786aca3805961514cf229482ad058674bdab WatchSource:0}: Error finding container 290d622a95caa3252d069d6b963e786aca3805961514cf229482ad058674bdab: Status 404 returned error can't find the container with id 290d622a95caa3252d069d6b963e786aca3805961514cf229482ad058674bdab Jul 02 19:08:18.693012 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.692768 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-gnf78" event={"ID":"da2642bb-e400-40ec-bded-5f145bbba018","Type":"ContainerStarted","Data":"290d622a95caa3252d069d6b963e786aca3805961514cf229482ad058674bdab"} Jul 02 19:08:18.696370 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.696315 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gqj2t" event={"ID":"54667ba4-1e72-488c-9fe3-53026b8d2df9","Type":"ContainerStarted","Data":"8fa29099865a364a1cb289d0e4c71221d5431101b3d53bc0ef884ce6fb9a097b"} Jul 02 19:08:18.701334 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.701298 2565 generic.go:356] "Generic (PLEG): container finished" podID="ce6ec4d4-1f88-40e4-b020-e1e137efd53a" containerID="f07d1a833956ae23acf18cd0c28d4dc99362ec40b4c4e2d3c677edd3c23f4147" exitCode=0 Jul 02 19:08:18.701924 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.701849 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgwb4" event={"ID":"ce6ec4d4-1f88-40e4-b020-e1e137efd53a","Type":"ContainerDied","Data":"f07d1a833956ae23acf18cd0c28d4dc99362ec40b4c4e2d3c677edd3c23f4147"} Jul 02 19:08:18.714374 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.714161 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gqj2t" podStartSLOduration=0.791520189 podStartE2EDuration="1.714142759s" totalImagesPullingTime="922.62257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:18.7137661 +0000 UTC m=+37.829117324" watchObservedRunningTime="2026-07-02 19:08:18.714142759 +0000 UTC m=+37.829493959" Jul 02 19:08:18.769901 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.769768 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xlvvc"] Jul 02 19:08:18.774621 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:08:18.774589 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa48a740_00cf_4f9c_815b_c96d531852de.slice/crio-95187f96cacfa0d49b1dbdfe0245412f7a85da6c0e03c966fbe346cb350fbffe WatchSource:0}: Error finding container 95187f96cacfa0d49b1dbdfe0245412f7a85da6c0e03c966fbe346cb350fbffe: Status 404 returned error can't find the container with id 95187f96cacfa0d49b1dbdfe0245412f7a85da6c0e03c966fbe346cb350fbffe Jul 02 19:08:18.789245 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.789178 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-5pcbl"] Jul 02 19:08:18.793845 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:08:18.793818 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39433d94_da63_4417_8cac_bfe000cc31dd.slice/crio-ef479a02f858ca8ffbc42398a3148f50f488d00d914da9d310d7ce8de4a9d873 WatchSource:0}: Error finding container ef479a02f858ca8ffbc42398a3148f50f488d00d914da9d310d7ce8de4a9d873: Status 404 returned error can't find the container with id ef479a02f858ca8ffbc42398a3148f50f488d00d914da9d310d7ce8de4a9d873 Jul 02 19:08:18.985329 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:18.985286 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d408f7b-881f-43f1-89a1-61bcd8cb5261-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-6cr9w\" (UID: \"5d408f7b-881f-43f1-89a1-61bcd8cb5261\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w" Jul 02 19:08:18.985519 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:18.985456 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 19:08:18.985519 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:18.985519 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d408f7b-881f-43f1-89a1-61bcd8cb5261-samples-operator-tls podName:5d408f7b-881f-43f1-89a1-61bcd8cb5261 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:20.985503476 +0000 UTC m=+40.100854687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5d408f7b-881f-43f1-89a1-61bcd8cb5261-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-6cr9w" (UID: "5d408f7b-881f-43f1-89a1-61bcd8cb5261") : secret "samples-operator-tls" not found Jul 02 19:08:19.704661 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:19.704618 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xlvvc" event={"ID":"fa48a740-00cf-4f9c-815b-c96d531852de","Type":"ContainerStarted","Data":"95187f96cacfa0d49b1dbdfe0245412f7a85da6c0e03c966fbe346cb350fbffe"} Jul 02 19:08:19.707722 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:19.707692 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dgwb4" event={"ID":"ce6ec4d4-1f88-40e4-b020-e1e137efd53a","Type":"ContainerStarted","Data":"d08b2dcdd762e0f9fb17771ff4a49b5d4b430088b71db2a4b4fedd88e4b03ab5"} Jul 02 19:08:19.708683 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:19.708661 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" event={"ID":"39433d94-da63-4417-8cac-bfe000cc31dd","Type":"ContainerStarted","Data":"ef479a02f858ca8ffbc42398a3148f50f488d00d914da9d310d7ce8de4a9d873"} Jul 02 19:08:19.736875 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:19.736818 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dgwb4" podStartSLOduration=8.949917942 podStartE2EDuration="38.736802142s" totalImagesPullingTime="29.7868842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:07:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:19.735534155 +0000 UTC m=+38.850885378" watchObservedRunningTime="2026-07-02 19:08:19.736802142 +0000 UTC m=+38.852153365" Jul 02 19:08:20.237554 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.236022 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2tmvt"] Jul 02 19:08:20.260913 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.259861 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2tmvt"] Jul 02 19:08:20.260913 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.260006 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:20.263413 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.263385 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 02 19:08:20.265143 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.264742 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 02 19:08:20.265143 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.264985 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jjnz9\"" Jul 02 19:08:20.265143 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.265067 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 02 19:08:20.266303 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.266129 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 02 19:08:20.397364 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.397150 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:20.397364 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.397222 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4996d3a4-2f90-403e-a277-df9983b8e7f0-crio-socket\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:20.397364 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.397329 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4996d3a4-2f90-403e-a277-df9983b8e7f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:20.397637 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.397392 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4996d3a4-2f90-403e-a277-df9983b8e7f0-data-volume\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:20.397637 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.397480 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djm6z\" (UniqueName: \"kubernetes.io/projected/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-api-access-djm6z\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:20.498752 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.498138 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djm6z\" (UniqueName: \"kubernetes.io/projected/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-api-access-djm6z\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:20.498752 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.498197 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:20.498752 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.498238 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls\") pod \"dns-default-8fcgj\" (UID: \"442ba05b-cbb8-48fb-b1ab-96ebe4960f80\") " pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:20.498752 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.498267 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4996d3a4-2f90-403e-a277-df9983b8e7f0-crio-socket\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:20.498752 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.498316 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4996d3a4-2f90-403e-a277-df9983b8e7f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:20.498752 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:20.498319 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 19:08:20.498752 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.498360 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4996d3a4-2f90-403e-a277-df9983b8e7f0-data-volume\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:20.498752 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:20.498417 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-rbac-proxy-cm podName:4996d3a4-2f90-403e-a277-df9983b8e7f0 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:20.99839426 +0000 UTC m=+40.113745539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2tmvt" (UID: "4996d3a4-2f90-403e-a277-df9983b8e7f0") : configmap "kube-rbac-proxy" not found Jul 02 19:08:20.498752 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:20.498696 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:08:20.498752 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:20.498722 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 19:08:20.498752 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.498727 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4996d3a4-2f90-403e-a277-df9983b8e7f0-crio-socket\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:20.498752 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:20.498740 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls podName:442ba05b-cbb8-48fb-b1ab-96ebe4960f80 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:24.498725936 +0000 UTC m=+43.614077141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls") pod "dns-default-8fcgj" (UID: "442ba05b-cbb8-48fb-b1ab-96ebe4960f80") : secret "dns-default-metrics-tls" not found Jul 02 19:08:20.498752 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.498726 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4996d3a4-2f90-403e-a277-df9983b8e7f0-data-volume\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:20.499522 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:20.498779 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4996d3a4-2f90-403e-a277-df9983b8e7f0-insights-runtime-extractor-tls podName:4996d3a4-2f90-403e-a277-df9983b8e7f0 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:20.998761874 +0000 UTC m=+40.114113079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4996d3a4-2f90-403e-a277-df9983b8e7f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2tmvt" (UID: "4996d3a4-2f90-403e-a277-df9983b8e7f0") : secret "insights-runtime-extractor-tls" not found Jul 02 19:08:20.512967 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.512936 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djm6z\" (UniqueName: \"kubernetes.io/projected/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-api-access-djm6z\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:20.714363 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.714263 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wlnv6" event={"ID":"e81d42fb-8cb6-4e07-863d-4912aa0ed5e8","Type":"ContainerStarted","Data":"50f485be8824e6c7c1763897e5f392dbdf442f50ff42a899dc57d0e1bc3c3a43"} Jul 02 19:08:20.736121 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:20.735783 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wlnv6" podStartSLOduration=1.907850411 podStartE2EDuration="4.735760273s" totalImagesPullingTime="2.827909862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:20.734932074 +0000 UTC m=+39.850283318" watchObservedRunningTime="2026-07-02 19:08:20.735760273 +0000 UTC m=+39.851111498" Jul 02 19:08:21.004269 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:21.003940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4996d3a4-2f90-403e-a277-df9983b8e7f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:21.004269 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:21.004000 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d408f7b-881f-43f1-89a1-61bcd8cb5261-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-6cr9w\" (UID: \"5d408f7b-881f-43f1-89a1-61bcd8cb5261\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w" Jul 02 19:08:21.004269 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:21.004083 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:21.004269 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:21.004113 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 19:08:21.004269 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:21.004173 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4996d3a4-2f90-403e-a277-df9983b8e7f0-insights-runtime-extractor-tls podName:4996d3a4-2f90-403e-a277-df9983b8e7f0 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:22.004157734 +0000 UTC m=+41.119508935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4996d3a4-2f90-403e-a277-df9983b8e7f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2tmvt" (UID: "4996d3a4-2f90-403e-a277-df9983b8e7f0") : secret "insights-runtime-extractor-tls" not found Jul 02 19:08:21.004269 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:21.004114 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 19:08:21.004269 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:21.004202 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 19:08:21.004269 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:21.004258 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d408f7b-881f-43f1-89a1-61bcd8cb5261-samples-operator-tls podName:5d408f7b-881f-43f1-89a1-61bcd8cb5261 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:25.004239625 +0000 UTC m=+44.119590831 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5d408f7b-881f-43f1-89a1-61bcd8cb5261-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-6cr9w" (UID: "5d408f7b-881f-43f1-89a1-61bcd8cb5261") : secret "samples-operator-tls" not found Jul 02 19:08:21.004791 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:21.004283 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-rbac-proxy-cm podName:4996d3a4-2f90-403e-a277-df9983b8e7f0 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:22.004271444 +0000 UTC m=+41.119622646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2tmvt" (UID: "4996d3a4-2f90-403e-a277-df9983b8e7f0") : configmap "kube-rbac-proxy" not found Jul 02 19:08:21.820686 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:21.820653 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dqx74_e9bceedd-6e94-4988-baf5-5eb9eb417a7b/dns-node-resolver/0.log" Jul 02 19:08:22.013054 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:22.013004 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4996d3a4-2f90-403e-a277-df9983b8e7f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:22.013250 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:22.013136 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:22.013250 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:22.013167 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 19:08:22.013392 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:22.013252 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4996d3a4-2f90-403e-a277-df9983b8e7f0-insights-runtime-extractor-tls podName:4996d3a4-2f90-403e-a277-df9983b8e7f0 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:24.013229484 +0000 UTC m=+43.128580690 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4996d3a4-2f90-403e-a277-df9983b8e7f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2tmvt" (UID: "4996d3a4-2f90-403e-a277-df9983b8e7f0") : secret "insights-runtime-extractor-tls" not found Jul 02 19:08:22.013392 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:22.013307 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 19:08:22.013392 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:22.013379 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-rbac-proxy-cm podName:4996d3a4-2f90-403e-a277-df9983b8e7f0 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:24.01336262 +0000 UTC m=+43.128713838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2tmvt" (UID: "4996d3a4-2f90-403e-a277-df9983b8e7f0") : configmap "kube-rbac-proxy" not found Jul 02 19:08:22.620897 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:22.620869 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hf8w8_b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39/node-ca/0.log" Jul 02 19:08:23.422002 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.421975 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wlnv6_e81d42fb-8cb6-4e07-863d-4912aa0ed5e8/serve-healthcheck-canary/0.log" Jul 02 19:08:23.671157 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.671127 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69c8bb978-vh2cc"] Jul 02 19:08:23.688265 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.688240 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.690928 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.690901 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 02 19:08:23.691443 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.691428 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 02 19:08:23.692020 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.691997 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 02 19:08:23.692172 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.692038 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-d4gw7\"" Jul 02 19:08:23.695676 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.695618 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69c8bb978-vh2cc"] Jul 02 19:08:23.697969 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.697939 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 02 19:08:23.723666 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.723641 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/0.log" Jul 02 19:08:23.723844 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.723682 2565 generic.go:356] "Generic (PLEG): container finished" podID="39433d94-da63-4417-8cac-bfe000cc31dd" containerID="b9101a1bafb6165ef1f5a41f16ba4ec09857b2e1780f075bf35400e51ff5835b" exitCode=255 Jul 02 19:08:23.723844 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.723716 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" event={"ID":"39433d94-da63-4417-8cac-bfe000cc31dd","Type":"ContainerDied","Data":"b9101a1bafb6165ef1f5a41f16ba4ec09857b2e1780f075bf35400e51ff5835b"} Jul 02 19:08:23.724034 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.723997 2565 scope.go:117] "RemoveContainer" containerID="b9101a1bafb6165ef1f5a41f16ba4ec09857b2e1780f075bf35400e51ff5835b" Jul 02 19:08:23.725458 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.725419 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-gnf78" event={"ID":"da2642bb-e400-40ec-bded-5f145bbba018","Type":"ContainerStarted","Data":"3fe6b82918f8267d6b7cd5bebeb354e8ec95d65a7f032fedfe92cdd3b7d52b28"} Jul 02 19:08:23.726951 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.726917 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xlvvc" event={"ID":"fa48a740-00cf-4f9c-815b-c96d531852de","Type":"ContainerStarted","Data":"031f3c33b7a211c7b6d4a0ac4141ea91d119668c6d423834f249bf4fb241ad0e"} Jul 02 19:08:23.763297 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.763236 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-gnf78" podStartSLOduration=0.880702096 podStartE2EDuration="5.763214737s" totalImagesPullingTime="4.882512641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:23.762768999 +0000 UTC m=+42.878120223" watchObservedRunningTime="2026-07-02 19:08:23.763214737 +0000 UTC m=+42.878565962" Jul 02 19:08:23.781169 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.781110 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xlvvc" podStartSLOduration=1.06407912 podStartE2EDuration="5.781091041s" totalImagesPullingTime="4.717011921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:23.780109557 +0000 UTC m=+42.895460778" watchObservedRunningTime="2026-07-02 19:08:23.781091041 +0000 UTC m=+42.896442266" Jul 02 19:08:23.828383 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.828333 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.829589 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.828570 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-bound-sa-token\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.829589 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.828857 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/964b1f60-6743-40ae-8ce3-3e69d1ed6181-installation-pull-secrets\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.829589 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.828894 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/964b1f60-6743-40ae-8ce3-3e69d1ed6181-trusted-ca\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.829589 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.829180 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-certificates\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.829589 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.829216 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/964b1f60-6743-40ae-8ce3-3e69d1ed6181-image-registry-private-configuration\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.829589 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.829260 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/964b1f60-6743-40ae-8ce3-3e69d1ed6181-ca-trust-extracted\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.829589 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.829490 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q9fb\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-kube-api-access-8q9fb\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.930050 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.930009 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/964b1f60-6743-40ae-8ce3-3e69d1ed6181-installation-pull-secrets\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.930050 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.930051 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/964b1f60-6743-40ae-8ce3-3e69d1ed6181-trusted-ca\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.930323 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.930241 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-certificates\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.930323 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.930283 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/964b1f60-6743-40ae-8ce3-3e69d1ed6181-image-registry-private-configuration\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.930323 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.930317 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/964b1f60-6743-40ae-8ce3-3e69d1ed6181-ca-trust-extracted\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.930491 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.930399 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q9fb\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-kube-api-access-8q9fb\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.930491 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.930435 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.930596 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.930524 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-bound-sa-token\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.931081 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.930695 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/964b1f60-6743-40ae-8ce3-3e69d1ed6181-ca-trust-extracted\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.931081 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:23.930784 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:08:23.931081 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:23.930807 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8bb978-vh2cc: secret "image-registry-tls" not found Jul 02 19:08:23.931081 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:23.930879 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls podName:964b1f60-6743-40ae-8ce3-3e69d1ed6181 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:24.430858645 +0000 UTC m=+43.546209851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls") pod "image-registry-69c8bb978-vh2cc" (UID: "964b1f60-6743-40ae-8ce3-3e69d1ed6181") : secret "image-registry-tls" not found Jul 02 19:08:23.931415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.931365 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/964b1f60-6743-40ae-8ce3-3e69d1ed6181-trusted-ca\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.931415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.931399 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-certificates\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.933029 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.933011 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/964b1f60-6743-40ae-8ce3-3e69d1ed6181-image-registry-private-configuration\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.933101 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.933035 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/964b1f60-6743-40ae-8ce3-3e69d1ed6181-installation-pull-secrets\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.939781 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.939753 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-bound-sa-token\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:23.940606 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:23.940584 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q9fb\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-kube-api-access-8q9fb\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:24.031335 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:24.031258 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:24.031335 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:24.031313 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4996d3a4-2f90-403e-a277-df9983b8e7f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:24.031502 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:24.031393 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 19:08:24.031502 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:24.031471 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-rbac-proxy-cm podName:4996d3a4-2f90-403e-a277-df9983b8e7f0 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:28.031454116 +0000 UTC m=+47.146805324 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2tmvt" (UID: "4996d3a4-2f90-403e-a277-df9983b8e7f0") : configmap "kube-rbac-proxy" not found Jul 02 19:08:24.031502 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:24.031468 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 19:08:24.031502 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:24.031498 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4996d3a4-2f90-403e-a277-df9983b8e7f0-insights-runtime-extractor-tls podName:4996d3a4-2f90-403e-a277-df9983b8e7f0 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:28.031492297 +0000 UTC m=+47.146843498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4996d3a4-2f90-403e-a277-df9983b8e7f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2tmvt" (UID: "4996d3a4-2f90-403e-a277-df9983b8e7f0") : secret "insights-runtime-extractor-tls" not found Jul 02 19:08:24.363085 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:24.363003 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-p4wp2"] Jul 02 19:08:24.398377 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:24.398327 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-p4wp2"] Jul 02 19:08:24.398553 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:24.398378 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-p4wp2" Jul 02 19:08:24.401206 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:24.401182 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-s4hwk\"" Jul 02 19:08:24.434510 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:24.434476 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:24.434870 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:24.434637 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:08:24.434870 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:24.434656 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8bb978-vh2cc: secret "image-registry-tls" not found Jul 02 19:08:24.434870 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:24.434724 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls podName:964b1f60-6743-40ae-8ce3-3e69d1ed6181 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:25.434703662 +0000 UTC m=+44.550054867 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls") pod "image-registry-69c8bb978-vh2cc" (UID: "964b1f60-6743-40ae-8ce3-3e69d1ed6181") : secret "image-registry-tls" not found Jul 02 19:08:24.534995 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:24.534964 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfknw\" (UniqueName: \"kubernetes.io/projected/3fbe7496-3b7f-4241-8cee-d1cfdda77fd9-kube-api-access-gfknw\") pod \"network-check-source-989f8b8dc-p4wp2\" (UID: \"3fbe7496-3b7f-4241-8cee-d1cfdda77fd9\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-p4wp2" Jul 02 19:08:24.535126 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:24.535030 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls\") pod \"dns-default-8fcgj\" (UID: \"442ba05b-cbb8-48fb-b1ab-96ebe4960f80\") " pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:24.535162 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:24.535142 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:08:24.535217 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:24.535206 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls podName:442ba05b-cbb8-48fb-b1ab-96ebe4960f80 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:32.535187175 +0000 UTC m=+51.650538376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls") pod "dns-default-8fcgj" (UID: "442ba05b-cbb8-48fb-b1ab-96ebe4960f80") : secret "dns-default-metrics-tls" not found Jul 02 19:08:24.635784 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:24.635688 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfknw\" (UniqueName: \"kubernetes.io/projected/3fbe7496-3b7f-4241-8cee-d1cfdda77fd9-kube-api-access-gfknw\") pod \"network-check-source-989f8b8dc-p4wp2\" (UID: \"3fbe7496-3b7f-4241-8cee-d1cfdda77fd9\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-p4wp2" Jul 02 19:08:24.644416 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:24.644389 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfknw\" (UniqueName: \"kubernetes.io/projected/3fbe7496-3b7f-4241-8cee-d1cfdda77fd9-kube-api-access-gfknw\") pod \"network-check-source-989f8b8dc-p4wp2\" (UID: \"3fbe7496-3b7f-4241-8cee-d1cfdda77fd9\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-p4wp2" Jul 02 19:08:24.707236 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:24.707195 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-p4wp2" Jul 02 19:08:24.731860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:24.731837 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:08:24.732298 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:24.732282 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/0.log" Jul 02 19:08:24.732411 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:24.732321 2565 generic.go:356] "Generic (PLEG): container finished" podID="39433d94-da63-4417-8cac-bfe000cc31dd" containerID="d219a587398978c24e2220591186dcee58f98062e46033a92081c223e00cdb81" exitCode=255 Jul 02 19:08:24.732411 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:24.732385 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" event={"ID":"39433d94-da63-4417-8cac-bfe000cc31dd","Type":"ContainerDied","Data":"d219a587398978c24e2220591186dcee58f98062e46033a92081c223e00cdb81"} Jul 02 19:08:24.732516 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:24.732449 2565 scope.go:117] "RemoveContainer" containerID="b9101a1bafb6165ef1f5a41f16ba4ec09857b2e1780f075bf35400e51ff5835b" Jul 02 19:08:24.732911 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:24.732893 2565 scope.go:117] "RemoveContainer" containerID="d219a587398978c24e2220591186dcee58f98062e46033a92081c223e00cdb81" Jul 02 19:08:24.733342 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:24.733071 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-5pcbl_openshift-console-operator(39433d94-da63-4417-8cac-bfe000cc31dd)\"" pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" podUID="39433d94-da63-4417-8cac-bfe000cc31dd" Jul 02 19:08:24.857570 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:24.857542 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-p4wp2"] Jul 02 19:08:24.860541 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:08:24.860517 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbe7496_3b7f_4241_8cee_d1cfdda77fd9.slice/crio-8e71d8e967ccf56655acbc8816a9d44a17f5faad12e658fa7e484617f4b8d4b7 WatchSource:0}: Error finding container 8e71d8e967ccf56655acbc8816a9d44a17f5faad12e658fa7e484617f4b8d4b7: Status 404 returned error can't find the container with id 8e71d8e967ccf56655acbc8816a9d44a17f5faad12e658fa7e484617f4b8d4b7 Jul 02 19:08:25.038234 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:25.038193 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d408f7b-881f-43f1-89a1-61bcd8cb5261-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-6cr9w\" (UID: \"5d408f7b-881f-43f1-89a1-61bcd8cb5261\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w" Jul 02 19:08:25.038457 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:25.038379 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 19:08:25.038457 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:25.038448 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d408f7b-881f-43f1-89a1-61bcd8cb5261-samples-operator-tls podName:5d408f7b-881f-43f1-89a1-61bcd8cb5261 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:33.038432045 +0000 UTC m=+52.153783246 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/5d408f7b-881f-43f1-89a1-61bcd8cb5261-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-6cr9w" (UID: "5d408f7b-881f-43f1-89a1-61bcd8cb5261") : secret "samples-operator-tls" not found Jul 02 19:08:25.442321 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:25.442275 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:25.442791 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:25.442447 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:08:25.442791 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:25.442464 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8bb978-vh2cc: secret "image-registry-tls" not found Jul 02 19:08:25.442791 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:25.442516 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls podName:964b1f60-6743-40ae-8ce3-3e69d1ed6181 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:27.442501647 +0000 UTC m=+46.557852851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls") pod "image-registry-69c8bb978-vh2cc" (UID: "964b1f60-6743-40ae-8ce3-3e69d1ed6181") : secret "image-registry-tls" not found Jul 02 19:08:25.736697 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:25.736618 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:08:25.738578 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:25.737477 2565 scope.go:117] "RemoveContainer" containerID="d219a587398978c24e2220591186dcee58f98062e46033a92081c223e00cdb81" Jul 02 19:08:25.738578 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:25.737757 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-5pcbl_openshift-console-operator(39433d94-da63-4417-8cac-bfe000cc31dd)\"" pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" podUID="39433d94-da63-4417-8cac-bfe000cc31dd" Jul 02 19:08:25.739031 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:25.739001 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-p4wp2" event={"ID":"3fbe7496-3b7f-4241-8cee-d1cfdda77fd9","Type":"ContainerStarted","Data":"8e71d8e967ccf56655acbc8816a9d44a17f5faad12e658fa7e484617f4b8d4b7"} Jul 02 19:08:26.950294 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:26.950257 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-k2vjv"] Jul 02 19:08:26.985264 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:26.985227 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-k2vjv"] Jul 02 19:08:26.985443 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:26.985376 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-k2vjv" Jul 02 19:08:26.988579 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:26.988549 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 02 19:08:26.989025 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:26.989002 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 02 19:08:26.989025 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:26.989016 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 02 19:08:26.990006 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:26.989965 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-g2dkb\"" Jul 02 19:08:26.990117 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:26.990012 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 02 19:08:27.157489 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:27.157452 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/424f7a98-4335-46a7-87ee-208624fa9b7b-signing-key\") pod \"service-ca-758f599c5-k2vjv\" (UID: \"424f7a98-4335-46a7-87ee-208624fa9b7b\") " pod="openshift-service-ca/service-ca-758f599c5-k2vjv" Jul 02 19:08:27.157652 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:27.157607 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/424f7a98-4335-46a7-87ee-208624fa9b7b-signing-cabundle\") pod \"service-ca-758f599c5-k2vjv\" (UID: \"424f7a98-4335-46a7-87ee-208624fa9b7b\") " pod="openshift-service-ca/service-ca-758f599c5-k2vjv" Jul 02 19:08:27.157652 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:27.157631 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjm74\" (UniqueName: \"kubernetes.io/projected/424f7a98-4335-46a7-87ee-208624fa9b7b-kube-api-access-bjm74\") pod \"service-ca-758f599c5-k2vjv\" (UID: \"424f7a98-4335-46a7-87ee-208624fa9b7b\") " pod="openshift-service-ca/service-ca-758f599c5-k2vjv" Jul 02 19:08:27.258958 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:27.258873 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/424f7a98-4335-46a7-87ee-208624fa9b7b-signing-key\") pod \"service-ca-758f599c5-k2vjv\" (UID: \"424f7a98-4335-46a7-87ee-208624fa9b7b\") " pod="openshift-service-ca/service-ca-758f599c5-k2vjv" Jul 02 19:08:27.259137 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:27.259108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/424f7a98-4335-46a7-87ee-208624fa9b7b-signing-cabundle\") pod \"service-ca-758f599c5-k2vjv\" (UID: \"424f7a98-4335-46a7-87ee-208624fa9b7b\") " pod="openshift-service-ca/service-ca-758f599c5-k2vjv" Jul 02 19:08:27.259205 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:27.259135 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjm74\" (UniqueName: \"kubernetes.io/projected/424f7a98-4335-46a7-87ee-208624fa9b7b-kube-api-access-bjm74\") pod \"service-ca-758f599c5-k2vjv\" (UID: \"424f7a98-4335-46a7-87ee-208624fa9b7b\") " pod="openshift-service-ca/service-ca-758f599c5-k2vjv" Jul 02 19:08:27.259704 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:27.259686 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/424f7a98-4335-46a7-87ee-208624fa9b7b-signing-cabundle\") pod \"service-ca-758f599c5-k2vjv\" (UID: \"424f7a98-4335-46a7-87ee-208624fa9b7b\") " pod="openshift-service-ca/service-ca-758f599c5-k2vjv" Jul 02 19:08:27.261363 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:27.261330 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/424f7a98-4335-46a7-87ee-208624fa9b7b-signing-key\") pod \"service-ca-758f599c5-k2vjv\" (UID: \"424f7a98-4335-46a7-87ee-208624fa9b7b\") " pod="openshift-service-ca/service-ca-758f599c5-k2vjv" Jul 02 19:08:27.267488 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:27.267469 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjm74\" (UniqueName: \"kubernetes.io/projected/424f7a98-4335-46a7-87ee-208624fa9b7b-kube-api-access-bjm74\") pod \"service-ca-758f599c5-k2vjv\" (UID: \"424f7a98-4335-46a7-87ee-208624fa9b7b\") " pod="openshift-service-ca/service-ca-758f599c5-k2vjv" Jul 02 19:08:27.296185 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:27.296153 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-k2vjv" Jul 02 19:08:27.435215 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:27.435136 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-k2vjv"] Jul 02 19:08:27.460791 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:27.460751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:27.460977 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:27.460902 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:08:27.460977 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:27.460923 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8bb978-vh2cc: secret "image-registry-tls" not found Jul 02 19:08:27.461098 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:27.460993 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls podName:964b1f60-6743-40ae-8ce3-3e69d1ed6181 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:31.460970401 +0000 UTC m=+50.576321604 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls") pod "image-registry-69c8bb978-vh2cc" (UID: "964b1f60-6743-40ae-8ce3-3e69d1ed6181") : secret "image-registry-tls" not found Jul 02 19:08:28.067006 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:28.066952 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4996d3a4-2f90-403e-a277-df9983b8e7f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:28.067467 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:28.067086 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 19:08:28.067467 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:28.067112 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:28.067467 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:28.067175 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4996d3a4-2f90-403e-a277-df9983b8e7f0-insights-runtime-extractor-tls podName:4996d3a4-2f90-403e-a277-df9983b8e7f0 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:36.067153762 +0000 UTC m=+55.182504964 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4996d3a4-2f90-403e-a277-df9983b8e7f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2tmvt" (UID: "4996d3a4-2f90-403e-a277-df9983b8e7f0") : secret "insights-runtime-extractor-tls" not found Jul 02 19:08:28.067467 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:28.067192 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 19:08:28.067467 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:28.067246 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-rbac-proxy-cm podName:4996d3a4-2f90-403e-a277-df9983b8e7f0 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:36.06722898 +0000 UTC m=+55.182580205 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2tmvt" (UID: "4996d3a4-2f90-403e-a277-df9983b8e7f0") : configmap "kube-rbac-proxy" not found Jul 02 19:08:28.454842 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:08:28.454807 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod424f7a98_4335_46a7_87ee_208624fa9b7b.slice/crio-b330fb31df41d1f8929e0156cc6e2b5b2a55fa77eb5d445cdde0558638c66e69 WatchSource:0}: Error finding container b330fb31df41d1f8929e0156cc6e2b5b2a55fa77eb5d445cdde0558638c66e69: Status 404 returned error can't find the container with id b330fb31df41d1f8929e0156cc6e2b5b2a55fa77eb5d445cdde0558638c66e69 Jul 02 19:08:28.572092 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:28.572062 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" Jul 02 19:08:28.572268 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:28.572122 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" Jul 02 19:08:28.572508 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:28.572496 2565 scope.go:117] "RemoveContainer" containerID="d219a587398978c24e2220591186dcee58f98062e46033a92081c223e00cdb81" Jul 02 19:08:28.572654 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:28.572636 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-5pcbl_openshift-console-operator(39433d94-da63-4417-8cac-bfe000cc31dd)\"" pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" podUID="39433d94-da63-4417-8cac-bfe000cc31dd" Jul 02 19:08:28.749459 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:28.749380 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-k2vjv" event={"ID":"424f7a98-4335-46a7-87ee-208624fa9b7b","Type":"ContainerStarted","Data":"37831d2354feea52e8dc06f387072c86f60ef42de21e91038af9d620d7794270"} Jul 02 19:08:28.749459 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:28.749431 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-k2vjv" event={"ID":"424f7a98-4335-46a7-87ee-208624fa9b7b","Type":"ContainerStarted","Data":"b330fb31df41d1f8929e0156cc6e2b5b2a55fa77eb5d445cdde0558638c66e69"} Jul 02 19:08:28.769768 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:28.769711 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-k2vjv" podStartSLOduration=2.769693524 podStartE2EDuration="2.769693524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:28.768530439 +0000 UTC m=+47.883881663" watchObservedRunningTime="2026-07-02 19:08:28.769693524 +0000 UTC m=+47.885044753" Jul 02 19:08:29.760644 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:29.760601 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-p4wp2" event={"ID":"3fbe7496-3b7f-4241-8cee-d1cfdda77fd9","Type":"ContainerStarted","Data":"0d80c4cb5bf71fe0b945c553f958483b31ce1da529d887bf38d23cd04c21b020"} Jul 02 19:08:29.778116 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:29.778048 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-p4wp2" podStartSLOduration=1.801075328 podStartE2EDuration="5.778027884s" totalImagesPullingTime="3.976952556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:29.777049118 +0000 UTC m=+48.892400370" watchObservedRunningTime="2026-07-02 19:08:29.778027884 +0000 UTC m=+48.893379108" Jul 02 19:08:31.499292 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:31.499250 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:31.499710 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:31.499421 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:08:31.499710 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:31.499441 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69c8bb978-vh2cc: secret "image-registry-tls" not found Jul 02 19:08:31.499710 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:31.499515 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls podName:964b1f60-6743-40ae-8ce3-3e69d1ed6181 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:39.49948018 +0000 UTC m=+58.614831384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls") pod "image-registry-69c8bb978-vh2cc" (UID: "964b1f60-6743-40ae-8ce3-3e69d1ed6181") : secret "image-registry-tls" not found Jul 02 19:08:32.609384 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:32.609331 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls\") pod \"dns-default-8fcgj\" (UID: \"442ba05b-cbb8-48fb-b1ab-96ebe4960f80\") " pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:32.609794 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:32.609511 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:08:32.609794 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:08:32.609589 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls podName:442ba05b-cbb8-48fb-b1ab-96ebe4960f80 nodeName:}" failed. No retries permitted until 2026-07-02 19:08:48.609566535 +0000 UTC m=+67.724917743 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls") pod "dns-default-8fcgj" (UID: "442ba05b-cbb8-48fb-b1ab-96ebe4960f80") : secret "dns-default-metrics-tls" not found Jul 02 19:08:33.112844 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:33.112787 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d408f7b-881f-43f1-89a1-61bcd8cb5261-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-6cr9w\" (UID: \"5d408f7b-881f-43f1-89a1-61bcd8cb5261\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w" Jul 02 19:08:33.115383 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:33.115331 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/5d408f7b-881f-43f1-89a1-61bcd8cb5261-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-6cr9w\" (UID: \"5d408f7b-881f-43f1-89a1-61bcd8cb5261\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w" Jul 02 19:08:33.146062 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:33.146021 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w" Jul 02 19:08:33.292889 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:33.292851 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w"] Jul 02 19:08:33.776371 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:33.776320 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w" event={"ID":"5d408f7b-881f-43f1-89a1-61bcd8cb5261","Type":"ContainerStarted","Data":"62762d1d27c7fa287978e2b2db3986bd25fa23ddb58c5b5b2bd1e6907c01f162"} Jul 02 19:08:35.785382 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:35.785273 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w" event={"ID":"5d408f7b-881f-43f1-89a1-61bcd8cb5261","Type":"ContainerStarted","Data":"251171316a6315f39b53e7a7f9116aae1f79e576f075e8b1c36c04d81c432a68"} Jul 02 19:08:35.785382 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:35.785339 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w" event={"ID":"5d408f7b-881f-43f1-89a1-61bcd8cb5261","Type":"ContainerStarted","Data":"0860417b52cc9f28afd8052f8b073a16ac88ea43a40c2c7d38114a7d33dbec7e"} Jul 02 19:08:36.139009 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:36.138905 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:36.139009 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:36.138961 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4996d3a4-2f90-403e-a277-df9983b8e7f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:36.139584 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:36.139556 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4996d3a4-2f90-403e-a277-df9983b8e7f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:36.141324 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:36.141304 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4996d3a4-2f90-403e-a277-df9983b8e7f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2tmvt\" (UID: \"4996d3a4-2f90-403e-a277-df9983b8e7f0\") " pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:36.176390 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:36.176315 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2tmvt" Jul 02 19:08:36.306457 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:36.306397 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-6cr9w" podStartSLOduration=17.215423696 podStartE2EDuration="19.306369403s" totalImagesPullingTime="2.090945707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:35.800562446 +0000 UTC m=+54.915913692" watchObservedRunningTime="2026-07-02 19:08:36.306369403 +0000 UTC m=+55.421720624" Jul 02 19:08:36.307061 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:36.307041 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2tmvt"] Jul 02 19:08:36.310009 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:08:36.309985 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4996d3a4_2f90_403e_a277_df9983b8e7f0.slice/crio-8647c3f23a8f76e5099443ec76f28e9814e8c7a4ad47770b6ac771407e679e0f WatchSource:0}: Error finding container 8647c3f23a8f76e5099443ec76f28e9814e8c7a4ad47770b6ac771407e679e0f: Status 404 returned error can't find the container with id 8647c3f23a8f76e5099443ec76f28e9814e8c7a4ad47770b6ac771407e679e0f Jul 02 19:08:36.792438 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:36.792385 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2tmvt" event={"ID":"4996d3a4-2f90-403e-a277-df9983b8e7f0","Type":"ContainerStarted","Data":"8647c3f23a8f76e5099443ec76f28e9814e8c7a4ad47770b6ac771407e679e0f"} Jul 02 19:08:39.573282 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:39.573226 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:39.576098 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:39.576066 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls\") pod \"image-registry-69c8bb978-vh2cc\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:39.645386 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:39.645331 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:40.228405 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:40.228368 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69c8bb978-vh2cc"] Jul 02 19:08:40.232829 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:08:40.232799 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964b1f60_6743_40ae_8ce3_3e69d1ed6181.slice/crio-f1465d076b4b018a4c94a0c106e11e987e7f05bac4da5917aef5f5a6f1d6d027 WatchSource:0}: Error finding container f1465d076b4b018a4c94a0c106e11e987e7f05bac4da5917aef5f5a6f1d6d027: Status 404 returned error can't find the container with id f1465d076b4b018a4c94a0c106e11e987e7f05bac4da5917aef5f5a6f1d6d027 Jul 02 19:08:40.656582 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:40.656552 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6bt54" Jul 02 19:08:40.808749 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:40.808709 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" event={"ID":"964b1f60-6743-40ae-8ce3-3e69d1ed6181","Type":"ContainerStarted","Data":"769a458f326adf35dfd9fc5d79d8f1db3d51790d64c35fb22ed32652a657a68e"} Jul 02 19:08:40.808924 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:40.808766 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" event={"ID":"964b1f60-6743-40ae-8ce3-3e69d1ed6181","Type":"ContainerStarted","Data":"f1465d076b4b018a4c94a0c106e11e987e7f05bac4da5917aef5f5a6f1d6d027"} Jul 02 19:08:40.808924 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:40.808795 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:08:40.810193 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:40.810170 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2tmvt" event={"ID":"4996d3a4-2f90-403e-a277-df9983b8e7f0","Type":"ContainerStarted","Data":"63e65f6bc2b71679ddba19df5ac54e4e8b6de04dba126d92ad4488048ce19ae4"} Jul 02 19:08:40.849195 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:40.849141 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" podStartSLOduration=17.849122943 podStartE2EDuration="17.849122943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:40.849055707 +0000 UTC m=+59.964406930" watchObservedRunningTime="2026-07-02 19:08:40.849122943 +0000 UTC m=+59.964474182" Jul 02 19:08:41.815977 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:41.815930 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2tmvt" event={"ID":"4996d3a4-2f90-403e-a277-df9983b8e7f0","Type":"ContainerStarted","Data":"7ca9f0025d3e64158d9c5b94a18fdf8239c1ea7b1e26977c7b88b6d4c0aea1de"} Jul 02 19:08:42.394628 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:42.394595 2565 scope.go:117] "RemoveContainer" containerID="d219a587398978c24e2220591186dcee58f98062e46033a92081c223e00cdb81" Jul 02 19:08:42.821207 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:42.821179 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:08:42.821695 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:42.821258 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" event={"ID":"39433d94-da63-4417-8cac-bfe000cc31dd","Type":"ContainerStarted","Data":"350ba92da3ddf92cc6d7af4c8d2868bfe3f63409bd4df83741c2c11e87529555"} Jul 02 19:08:42.821695 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:42.821569 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" Jul 02 19:08:42.839461 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:42.839401 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" podStartSLOduration=20.141654835 podStartE2EDuration="24.839383663s" totalImagesPullingTime="4.697728828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:42.838996031 +0000 UTC m=+61.954347254" watchObservedRunningTime="2026-07-02 19:08:42.839383663 +0000 UTC m=+61.954734885" Jul 02 19:08:42.896907 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:42.896875 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-5pcbl" Jul 02 19:08:44.829601 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:44.829565 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2tmvt" event={"ID":"4996d3a4-2f90-403e-a277-df9983b8e7f0","Type":"ContainerStarted","Data":"c5454d78b9f770d07934bf55b9f0a7d823b3a32271debfe0237d9bf27dc9225f"} Jul 02 19:08:44.871165 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:44.871108 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2tmvt" podStartSLOduration=17.509080088 podStartE2EDuration="24.871092661s" totalImagesPullingTime="7.362012573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:08:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:44.870806112 +0000 UTC m=+63.986157335" watchObservedRunningTime="2026-07-02 19:08:44.871092661 +0000 UTC m=+63.986443883" Jul 02 19:08:46.133222 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.133181 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret\") pod \"global-pull-secret-syncer-jfl78\" (UID: \"f714a87f-b1c3-4e57-8a2f-c1f61ab315be\") " pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:08:46.135975 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.135955 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 19:08:46.146263 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.146225 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714a87f-b1c3-4e57-8a2f-c1f61ab315be-original-pull-secret\") pod \"global-pull-secret-syncer-jfl78\" (UID: \"f714a87f-b1c3-4e57-8a2f-c1f61ab315be\") " pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:08:46.209771 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.209745 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfl78" Jul 02 19:08:46.234573 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.234537 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhr4b\" (UniqueName: \"kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b\") pod \"network-check-target-m66fc\" (UID: \"b1b2b08c-8ac4-431a-92df-15eb9b42cc4b\") " pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:08:46.234742 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.234606 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs\") pod \"network-metrics-daemon-ld8fm\" (UID: \"e21bc13b-a1b4-4934-9f3e-7eb280915f9b\") " pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:08:46.237287 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.237256 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 19:08:46.237750 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.237262 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhr4b\" (UniqueName: \"kubernetes.io/projected/b1b2b08c-8ac4-431a-92df-15eb9b42cc4b-kube-api-access-jhr4b\") pod \"network-check-target-m66fc\" (UID: \"b1b2b08c-8ac4-431a-92df-15eb9b42cc4b\") " pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:08:46.247709 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.247681 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e21bc13b-a1b4-4934-9f3e-7eb280915f9b-metrics-certs\") pod \"network-metrics-daemon-ld8fm\" (UID: \"e21bc13b-a1b4-4934-9f3e-7eb280915f9b\") " pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:08:46.342902 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.342867 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jfl78"] Jul 02 19:08:46.345776 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:08:46.345749 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf714a87f_b1c3_4e57_8a2f_c1f61ab315be.slice/crio-6bebf7b558b12f53a3e0af59a90a22bd4135c1ce163f4aaa4cfc17ec091fce21 WatchSource:0}: Error finding container 6bebf7b558b12f53a3e0af59a90a22bd4135c1ce163f4aaa4cfc17ec091fce21: Status 404 returned error can't find the container with id 6bebf7b558b12f53a3e0af59a90a22bd4135c1ce163f4aaa4cfc17ec091fce21 Jul 02 19:08:46.509898 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.509866 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r7xgd\"" Jul 02 19:08:46.514418 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.514397 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ld8fm" Jul 02 19:08:46.517132 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.517114 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5rf5h\"" Jul 02 19:08:46.524684 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.524652 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:08:46.655083 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.655017 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ld8fm"] Jul 02 19:08:46.657670 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:08:46.657645 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21bc13b_a1b4_4934_9f3e_7eb280915f9b.slice/crio-c46451ee98bbfc517561ca244dc91c6d10a3b7102f712976ae545d8077d884da WatchSource:0}: Error finding container c46451ee98bbfc517561ca244dc91c6d10a3b7102f712976ae545d8077d884da: Status 404 returned error can't find the container with id c46451ee98bbfc517561ca244dc91c6d10a3b7102f712976ae545d8077d884da Jul 02 19:08:46.675101 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.674961 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m66fc"] Jul 02 19:08:46.677476 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:08:46.677450 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b2b08c_8ac4_431a_92df_15eb9b42cc4b.slice/crio-cd6160153bbb8749c26b98f4ce783982ec7fb82acafa8f63fece3e98a6e6b8ee WatchSource:0}: Error finding container cd6160153bbb8749c26b98f4ce783982ec7fb82acafa8f63fece3e98a6e6b8ee: Status 404 returned error can't find the container with id cd6160153bbb8749c26b98f4ce783982ec7fb82acafa8f63fece3e98a6e6b8ee Jul 02 19:08:46.838909 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.838816 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m66fc" event={"ID":"b1b2b08c-8ac4-431a-92df-15eb9b42cc4b","Type":"ContainerStarted","Data":"fd2dd70b6cc7f006258fca116b4ef9d634c5f8ecde99b0add5620308f9fd7c00"} Jul 02 19:08:46.838909 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.838883 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m66fc" event={"ID":"b1b2b08c-8ac4-431a-92df-15eb9b42cc4b","Type":"ContainerStarted","Data":"cd6160153bbb8749c26b98f4ce783982ec7fb82acafa8f63fece3e98a6e6b8ee"} Jul 02 19:08:46.839117 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.838937 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:08:46.840308 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.840278 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ld8fm" event={"ID":"e21bc13b-a1b4-4934-9f3e-7eb280915f9b","Type":"ContainerStarted","Data":"c46451ee98bbfc517561ca244dc91c6d10a3b7102f712976ae545d8077d884da"} Jul 02 19:08:46.841443 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.841420 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jfl78" event={"ID":"f714a87f-b1c3-4e57-8a2f-c1f61ab315be","Type":"ContainerStarted","Data":"6bebf7b558b12f53a3e0af59a90a22bd4135c1ce163f4aaa4cfc17ec091fce21"} Jul 02 19:08:46.859274 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:46.859226 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m66fc" podStartSLOduration=65.859210678 podStartE2EDuration="1m5.859210678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:46.857891899 +0000 UTC m=+65.973243119" watchObservedRunningTime="2026-07-02 19:08:46.859210678 +0000 UTC m=+65.974561901" Jul 02 19:08:47.847568 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:47.847531 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ld8fm" event={"ID":"e21bc13b-a1b4-4934-9f3e-7eb280915f9b","Type":"ContainerStarted","Data":"e6826908e2bed14a395f4d8688702734cdf52e1bf7e3c24a7dc595e9c7a55834"} Jul 02 19:08:47.995778 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:47.995738 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69c8bb978-vh2cc"] Jul 02 19:08:48.019461 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.019426 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7755d6df84-m4s64"] Jul 02 19:08:48.037770 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.037739 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7755d6df84-m4s64" Jul 02 19:08:48.042595 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.042532 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8599677778-drclb"] Jul 02 19:08:48.044319 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.044273 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 02 19:08:48.044739 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.044713 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 02 19:08:48.045993 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.045971 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 02 19:08:48.046232 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.046216 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 02 19:08:48.056504 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.056475 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7755d6df84-m4s64"] Jul 02 19:08:48.056670 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.056617 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8599677778-drclb" Jul 02 19:08:48.061081 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.061052 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 02 19:08:48.061203 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.061146 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-75c5p\"" Jul 02 19:08:48.065684 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.065659 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8599677778-drclb"] Jul 02 19:08:48.118935 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.118901 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-544bbd9b4c-lw6hf"] Jul 02 19:08:48.136057 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.136024 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.149901 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.149871 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-544bbd9b4c-lw6hf"] Jul 02 19:08:48.153971 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.151121 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d4866b5-b270-481c-9b63-8737a9026088-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8599677778-drclb\" (UID: \"0d4866b5-b270-481c-9b63-8737a9026088\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8599677778-drclb" Jul 02 19:08:48.153971 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.151209 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca9df4c9-761a-4e01-9de0-668c3f6dd100-tmp\") pod \"klusterlet-addon-workmgr-7755d6df84-m4s64\" (UID: \"ca9df4c9-761a-4e01-9de0-668c3f6dd100\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7755d6df84-m4s64" Jul 02 19:08:48.153971 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.151268 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m7br\" (UniqueName: \"kubernetes.io/projected/0d4866b5-b270-481c-9b63-8737a9026088-kube-api-access-7m7br\") pod \"managed-serviceaccount-addon-agent-8599677778-drclb\" (UID: \"0d4866b5-b270-481c-9b63-8737a9026088\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8599677778-drclb" Jul 02 19:08:48.153971 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.151305 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpqng\" (UniqueName: \"kubernetes.io/projected/ca9df4c9-761a-4e01-9de0-668c3f6dd100-kube-api-access-mpqng\") pod \"klusterlet-addon-workmgr-7755d6df84-m4s64\" (UID: \"ca9df4c9-761a-4e01-9de0-668c3f6dd100\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7755d6df84-m4s64" Jul 02 19:08:48.153971 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.151358 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ca9df4c9-761a-4e01-9de0-668c3f6dd100-klusterlet-config\") pod \"klusterlet-addon-workmgr-7755d6df84-m4s64\" (UID: \"ca9df4c9-761a-4e01-9de0-668c3f6dd100\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7755d6df84-m4s64" Jul 02 19:08:48.252665 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.252568 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m7br\" (UniqueName: \"kubernetes.io/projected/0d4866b5-b270-481c-9b63-8737a9026088-kube-api-access-7m7br\") pod \"managed-serviceaccount-addon-agent-8599677778-drclb\" (UID: \"0d4866b5-b270-481c-9b63-8737a9026088\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8599677778-drclb" Jul 02 19:08:48.252665 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.252632 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/043ef48e-d5b2-4090-af93-39a4a503f8d1-image-registry-private-configuration\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.252913 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.252672 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqng\" (UniqueName: \"kubernetes.io/projected/ca9df4c9-761a-4e01-9de0-668c3f6dd100-kube-api-access-mpqng\") pod \"klusterlet-addon-workmgr-7755d6df84-m4s64\" (UID: \"ca9df4c9-761a-4e01-9de0-668c3f6dd100\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7755d6df84-m4s64" Jul 02 19:08:48.252913 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.252697 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/043ef48e-d5b2-4090-af93-39a4a503f8d1-registry-tls\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.252913 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.252733 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/043ef48e-d5b2-4090-af93-39a4a503f8d1-ca-trust-extracted\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.252913 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.252836 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/043ef48e-d5b2-4090-af93-39a4a503f8d1-registry-certificates\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.252913 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.252899 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ca9df4c9-761a-4e01-9de0-668c3f6dd100-klusterlet-config\") pod \"klusterlet-addon-workmgr-7755d6df84-m4s64\" (UID: \"ca9df4c9-761a-4e01-9de0-668c3f6dd100\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7755d6df84-m4s64" Jul 02 19:08:48.253135 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.252929 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/043ef48e-d5b2-4090-af93-39a4a503f8d1-installation-pull-secrets\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.253135 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.252951 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5g2f\" (UniqueName: \"kubernetes.io/projected/043ef48e-d5b2-4090-af93-39a4a503f8d1-kube-api-access-h5g2f\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.253135 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.252983 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/043ef48e-d5b2-4090-af93-39a4a503f8d1-trusted-ca\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.253135 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.253053 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d4866b5-b270-481c-9b63-8737a9026088-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8599677778-drclb\" (UID: \"0d4866b5-b270-481c-9b63-8737a9026088\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8599677778-drclb" Jul 02 19:08:48.253135 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.253114 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/043ef48e-d5b2-4090-af93-39a4a503f8d1-bound-sa-token\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.253328 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.253159 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca9df4c9-761a-4e01-9de0-668c3f6dd100-tmp\") pod \"klusterlet-addon-workmgr-7755d6df84-m4s64\" (UID: \"ca9df4c9-761a-4e01-9de0-668c3f6dd100\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7755d6df84-m4s64" Jul 02 19:08:48.253573 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.253536 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ca9df4c9-761a-4e01-9de0-668c3f6dd100-tmp\") pod \"klusterlet-addon-workmgr-7755d6df84-m4s64\" (UID: \"ca9df4c9-761a-4e01-9de0-668c3f6dd100\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7755d6df84-m4s64" Jul 02 19:08:48.255969 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.255941 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ca9df4c9-761a-4e01-9de0-668c3f6dd100-klusterlet-config\") pod \"klusterlet-addon-workmgr-7755d6df84-m4s64\" (UID: \"ca9df4c9-761a-4e01-9de0-668c3f6dd100\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7755d6df84-m4s64" Jul 02 19:08:48.256094 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.256070 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d4866b5-b270-481c-9b63-8737a9026088-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8599677778-drclb\" (UID: \"0d4866b5-b270-481c-9b63-8737a9026088\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8599677778-drclb" Jul 02 19:08:48.266612 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.266579 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m7br\" (UniqueName: \"kubernetes.io/projected/0d4866b5-b270-481c-9b63-8737a9026088-kube-api-access-7m7br\") pod \"managed-serviceaccount-addon-agent-8599677778-drclb\" (UID: \"0d4866b5-b270-481c-9b63-8737a9026088\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8599677778-drclb" Jul 02 19:08:48.266739 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.266680 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpqng\" (UniqueName: \"kubernetes.io/projected/ca9df4c9-761a-4e01-9de0-668c3f6dd100-kube-api-access-mpqng\") pod \"klusterlet-addon-workmgr-7755d6df84-m4s64\" (UID: \"ca9df4c9-761a-4e01-9de0-668c3f6dd100\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7755d6df84-m4s64" Jul 02 19:08:48.351943 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.351904 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7755d6df84-m4s64" Jul 02 19:08:48.354231 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.354203 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5g2f\" (UniqueName: \"kubernetes.io/projected/043ef48e-d5b2-4090-af93-39a4a503f8d1-kube-api-access-h5g2f\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.354374 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.354246 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/043ef48e-d5b2-4090-af93-39a4a503f8d1-trusted-ca\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.354374 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.354295 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/043ef48e-d5b2-4090-af93-39a4a503f8d1-bound-sa-token\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.354374 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.354343 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/043ef48e-d5b2-4090-af93-39a4a503f8d1-image-registry-private-configuration\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.354516 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.354396 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/043ef48e-d5b2-4090-af93-39a4a503f8d1-registry-tls\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.354516 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.354422 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/043ef48e-d5b2-4090-af93-39a4a503f8d1-ca-trust-extracted\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.354516 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.354445 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/043ef48e-d5b2-4090-af93-39a4a503f8d1-registry-certificates\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.354516 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.354487 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/043ef48e-d5b2-4090-af93-39a4a503f8d1-installation-pull-secrets\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.354992 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.354941 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/043ef48e-d5b2-4090-af93-39a4a503f8d1-ca-trust-extracted\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.355385 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.355310 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/043ef48e-d5b2-4090-af93-39a4a503f8d1-registry-certificates\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.355569 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.355550 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/043ef48e-d5b2-4090-af93-39a4a503f8d1-trusted-ca\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.357685 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.357628 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/043ef48e-d5b2-4090-af93-39a4a503f8d1-image-registry-private-configuration\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.357970 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.357923 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/043ef48e-d5b2-4090-af93-39a4a503f8d1-installation-pull-secrets\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.358074 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.358048 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/043ef48e-d5b2-4090-af93-39a4a503f8d1-registry-tls\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.363640 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.363612 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5g2f\" (UniqueName: \"kubernetes.io/projected/043ef48e-d5b2-4090-af93-39a4a503f8d1-kube-api-access-h5g2f\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.364055 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.364031 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/043ef48e-d5b2-4090-af93-39a4a503f8d1-bound-sa-token\") pod \"image-registry-544bbd9b4c-lw6hf\" (UID: \"043ef48e-d5b2-4090-af93-39a4a503f8d1\") " pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.377223 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.377183 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8599677778-drclb" Jul 02 19:08:48.452010 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.451732 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.577384 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.577055 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7755d6df84-m4s64"] Jul 02 19:08:48.582014 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:08:48.581982 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9df4c9_761a_4e01_9de0_668c3f6dd100.slice/crio-99420bbe63320f034e3caa152c4fc6eb00907c164bee0c69becc9a819fe6ee1d WatchSource:0}: Error finding container 99420bbe63320f034e3caa152c4fc6eb00907c164bee0c69becc9a819fe6ee1d: Status 404 returned error can't find the container with id 99420bbe63320f034e3caa152c4fc6eb00907c164bee0c69becc9a819fe6ee1d Jul 02 19:08:48.601920 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.601856 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8599677778-drclb"] Jul 02 19:08:48.605884 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:08:48.605851 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d4866b5_b270_481c_9b63_8737a9026088.slice/crio-55d10cb8e1fa140d47b2df799c26e9fda8223914b6c712e5fa8ed075d1f5ce26 WatchSource:0}: Error finding container 55d10cb8e1fa140d47b2df799c26e9fda8223914b6c712e5fa8ed075d1f5ce26: Status 404 returned error can't find the container with id 55d10cb8e1fa140d47b2df799c26e9fda8223914b6c712e5fa8ed075d1f5ce26 Jul 02 19:08:48.658756 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.658528 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-544bbd9b4c-lw6hf"] Jul 02 19:08:48.658847 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.658816 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls\") pod \"dns-default-8fcgj\" (UID: \"442ba05b-cbb8-48fb-b1ab-96ebe4960f80\") " pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:48.661288 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.661262 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/442ba05b-cbb8-48fb-b1ab-96ebe4960f80-metrics-tls\") pod \"dns-default-8fcgj\" (UID: \"442ba05b-cbb8-48fb-b1ab-96ebe4960f80\") " pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:48.662338 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:08:48.662305 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043ef48e_d5b2_4090_af93_39a4a503f8d1.slice/crio-d70175c7e9e61965253b5098c2b3ff6d92955bf651692b53365e11cd19034acc WatchSource:0}: Error finding container d70175c7e9e61965253b5098c2b3ff6d92955bf651692b53365e11cd19034acc: Status 404 returned error can't find the container with id d70175c7e9e61965253b5098c2b3ff6d92955bf651692b53365e11cd19034acc Jul 02 19:08:48.857686 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.857182 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" event={"ID":"043ef48e-d5b2-4090-af93-39a4a503f8d1","Type":"ContainerStarted","Data":"94fc152fa85186bf967fa744cc62beda4b13a34b50ecfdf1e4f0f499b9e38134"} Jul 02 19:08:48.857686 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.857246 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" event={"ID":"043ef48e-d5b2-4090-af93-39a4a503f8d1","Type":"ContainerStarted","Data":"d70175c7e9e61965253b5098c2b3ff6d92955bf651692b53365e11cd19034acc"} Jul 02 19:08:48.857686 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.857294 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:08:48.859036 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.859001 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8599677778-drclb" event={"ID":"0d4866b5-b270-481c-9b63-8737a9026088","Type":"ContainerStarted","Data":"55d10cb8e1fa140d47b2df799c26e9fda8223914b6c712e5fa8ed075d1f5ce26"} Jul 02 19:08:48.860493 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.860371 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7755d6df84-m4s64" event={"ID":"ca9df4c9-761a-4e01-9de0-668c3f6dd100","Type":"ContainerStarted","Data":"99420bbe63320f034e3caa152c4fc6eb00907c164bee0c69becc9a819fe6ee1d"} Jul 02 19:08:48.862598 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.862573 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ld8fm" event={"ID":"e21bc13b-a1b4-4934-9f3e-7eb280915f9b","Type":"ContainerStarted","Data":"ad04d1adec085b1dbdacffb98fe9bb5d0cb95a936dac9c33ab564eea2e19eb97"} Jul 02 19:08:48.877090 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.877023 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-45sjn\"" Jul 02 19:08:48.878685 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.878613 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" podStartSLOduration=0.878598094 podStartE2EDuration="878.598094ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:48.877220353 +0000 UTC m=+67.992571612" watchObservedRunningTime="2026-07-02 19:08:48.878598094 +0000 UTC m=+67.993949371" Jul 02 19:08:48.884408 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.884380 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:48.893990 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:48.893924 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ld8fm" podStartSLOduration=66.934183889 podStartE2EDuration="1m7.893901835s" totalImagesPullingTime="959.717946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:48.892083541 +0000 UTC m=+68.007434765" watchObservedRunningTime="2026-07-02 19:08:48.893901835 +0000 UTC m=+68.009253058" Jul 02 19:08:50.386924 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:50.386900 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8fcgj"] Jul 02 19:08:50.388492 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:08:50.388463 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442ba05b_cbb8_48fb_b1ab_96ebe4960f80.slice/crio-fc146028314dd831170e305ecefd667224d65521071d338c84563d961861c014 WatchSource:0}: Error finding container fc146028314dd831170e305ecefd667224d65521071d338c84563d961861c014: Status 404 returned error can't find the container with id fc146028314dd831170e305ecefd667224d65521071d338c84563d961861c014 Jul 02 19:08:50.885871 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:50.876288 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jfl78" event={"ID":"f714a87f-b1c3-4e57-8a2f-c1f61ab315be","Type":"ContainerStarted","Data":"ec057fec15325b76f158f41008500d54eccb3b5c6cf2d334fbbf62a1a3eaddc0"} Jul 02 19:08:50.885871 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:50.878132 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8fcgj" event={"ID":"442ba05b-cbb8-48fb-b1ab-96ebe4960f80","Type":"ContainerStarted","Data":"fc146028314dd831170e305ecefd667224d65521071d338c84563d961861c014"} Jul 02 19:08:53.205969 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.205896 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jfl78" podStartSLOduration=67.924336242 podStartE2EDuration="1m12.20587633s" totalImagesPullingTime="4.281540088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:50.895948391 +0000 UTC m=+70.011299614" watchObservedRunningTime="2026-07-02 19:08:53.20587633 +0000 UTC m=+72.321227553" Jul 02 19:08:53.206795 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.206771 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-844fb7b7bf-npslr"] Jul 02 19:08:53.209331 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.209310 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:53.212341 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.212292 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 02 19:08:53.212473 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.212442 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 02 19:08:53.213297 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.213276 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 02 19:08:53.213428 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.213384 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 02 19:08:53.213661 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.213639 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 02 19:08:53.213752 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.213664 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kq4f6\"" Jul 02 19:08:53.213752 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.213645 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 02 19:08:53.213864 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.213645 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 02 19:08:53.224989 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.224962 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-844fb7b7bf-npslr"] Jul 02 19:08:53.300507 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.300472 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-console-config\") pod \"console-844fb7b7bf-npslr\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:53.300708 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.300521 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-console-oauth-config\") pod \"console-844fb7b7bf-npslr\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:53.300708 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.300608 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vfwz\" (UniqueName: \"kubernetes.io/projected/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-kube-api-access-9vfwz\") pod \"console-844fb7b7bf-npslr\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:53.300708 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.300666 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-oauth-serving-cert\") pod \"console-844fb7b7bf-npslr\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:53.300708 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.300695 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-console-serving-cert\") pod \"console-844fb7b7bf-npslr\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:53.300875 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.300717 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-service-ca\") pod \"console-844fb7b7bf-npslr\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:53.401823 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.401785 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-console-config\") pod \"console-844fb7b7bf-npslr\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:53.402007 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.401840 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-console-oauth-config\") pod \"console-844fb7b7bf-npslr\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:53.402007 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.401873 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vfwz\" (UniqueName: \"kubernetes.io/projected/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-kube-api-access-9vfwz\") pod \"console-844fb7b7bf-npslr\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:53.402007 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.401898 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-oauth-serving-cert\") pod \"console-844fb7b7bf-npslr\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:53.402007 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.401916 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-console-serving-cert\") pod \"console-844fb7b7bf-npslr\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:53.402223 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.402132 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-service-ca\") pod \"console-844fb7b7bf-npslr\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:53.402733 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.402710 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-oauth-serving-cert\") pod \"console-844fb7b7bf-npslr\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:53.402850 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.402717 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-console-config\") pod \"console-844fb7b7bf-npslr\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:53.402850 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.402804 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-service-ca\") pod \"console-844fb7b7bf-npslr\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:53.405055 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.405031 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-console-serving-cert\") pod \"console-844fb7b7bf-npslr\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:53.405148 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.405042 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-console-oauth-config\") pod \"console-844fb7b7bf-npslr\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:53.412040 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.411994 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vfwz\" (UniqueName: \"kubernetes.io/projected/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-kube-api-access-9vfwz\") pod \"console-844fb7b7bf-npslr\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:53.520565 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:53.520464 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:08:54.655007 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:54.654956 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-844fb7b7bf-npslr"] Jul 02 19:08:54.659442 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:08:54.659406 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc5a6aa_acdb_457b_aeca_59d5a20f09fb.slice/crio-1a19e35a6cb7a65c8616e7dbca9eb876ca72ff3bc741a9b0909fa29b16f51833 WatchSource:0}: Error finding container 1a19e35a6cb7a65c8616e7dbca9eb876ca72ff3bc741a9b0909fa29b16f51833: Status 404 returned error can't find the container with id 1a19e35a6cb7a65c8616e7dbca9eb876ca72ff3bc741a9b0909fa29b16f51833 Jul 02 19:08:54.896754 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:54.896713 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8fcgj" event={"ID":"442ba05b-cbb8-48fb-b1ab-96ebe4960f80","Type":"ContainerStarted","Data":"cb73e9b45d0ad4a4474a48ceb7b84ed73933baa9ed17e097f7df3dd74990416b"} Jul 02 19:08:54.896959 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:54.896826 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8fcgj" Jul 02 19:08:54.896959 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:54.896855 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8fcgj" event={"ID":"442ba05b-cbb8-48fb-b1ab-96ebe4960f80","Type":"ContainerStarted","Data":"5b1f78457c47169dda4afd16e347e5f9922636c55c3647e6ae786072bca27d28"} Jul 02 19:08:54.898279 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:54.898255 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8599677778-drclb" event={"ID":"0d4866b5-b270-481c-9b63-8737a9026088","Type":"ContainerStarted","Data":"1147766ab54a8739bb2a5b4366370a6358d404c26ab23fbe79ced196e5b0c807"} Jul 02 19:08:54.899754 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:54.899729 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7755d6df84-m4s64" event={"ID":"ca9df4c9-761a-4e01-9de0-668c3f6dd100","Type":"ContainerStarted","Data":"8154418cc7fb2ebe5a205712a950c64754d84668002d03c63f9041ba7886aa37"} Jul 02 19:08:54.899870 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:54.899848 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7755d6df84-m4s64" Jul 02 19:08:54.900947 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:54.900922 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-844fb7b7bf-npslr" event={"ID":"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb","Type":"ContainerStarted","Data":"1a19e35a6cb7a65c8616e7dbca9eb876ca72ff3bc741a9b0909fa29b16f51833"} Jul 02 19:08:54.901596 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:54.901570 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7755d6df84-m4s64" Jul 02 19:08:54.916791 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:54.916741 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8fcgj" podStartSLOduration=34.815010438 podStartE2EDuration="38.916727206s" totalImagesPullingTime="4.101716768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:54.91551464 +0000 UTC m=+74.030865866" watchObservedRunningTime="2026-07-02 19:08:54.916727206 +0000 UTC m=+74.032078429" Jul 02 19:08:54.951966 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:54.951911 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7755d6df84-m4s64" podStartSLOduration=2.030945165 podStartE2EDuration="7.951894276s" totalImagesPullingTime="5.920949111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:54.951513495 +0000 UTC m=+74.066864718" watchObservedRunningTime="2026-07-02 19:08:54.951894276 +0000 UTC m=+74.067245501" Jul 02 19:08:54.952701 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:54.952666 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8599677778-drclb" podStartSLOduration=1.070595493 podStartE2EDuration="6.952654562s" totalImagesPullingTime="5.882059069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:54.932097426 +0000 UTC m=+74.047448648" watchObservedRunningTime="2026-07-02 19:08:54.952654562 +0000 UTC m=+74.068005784" Jul 02 19:08:58.003153 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:58.003113 2565 patch_prober.go:28] interesting pod/image-registry-69c8bb978-vh2cc container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 02 19:08:58.003641 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:58.003178 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" podUID="964b1f60-6743-40ae-8ce3-3e69d1ed6181" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:08:59.923010 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:59.922969 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-844fb7b7bf-npslr" event={"ID":"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb","Type":"ContainerStarted","Data":"6c5a8e70618545ab23ddc34414a74b2c4bcabef830a38b0a5eb8a1446de0b351"} Jul 02 19:08:59.942056 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:08:59.941985 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-844fb7b7bf-npslr" podStartSLOduration=2.643818445 podStartE2EDuration="6.941963418s" totalImagesPullingTime="4.298144973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:59.940544422 +0000 UTC m=+79.055895631" watchObservedRunningTime="2026-07-02 19:08:59.941963418 +0000 UTC m=+79.057314643" Jul 02 19:09:03.520820 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:03.520708 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:09:03.520820 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:03.520768 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:09:03.522062 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:03.522040 2565 patch_prober.go:28] interesting pod/console-844fb7b7bf-npslr container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.20:8443/health\": dial tcp 10.133.0.20:8443: connect: connection refused" start-of-body= Jul 02 19:09:03.522124 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:03.522082 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-844fb7b7bf-npslr" podUID="cbc5a6aa-acdb-457b-aeca-59d5a20f09fb" containerName="console" probeResult="failure" output="Get \"https://10.133.0.20:8443/health\": dial tcp 10.133.0.20:8443: connect: connection refused" Jul 02 19:09:04.906174 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:04.906139 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8fcgj" Jul 02 19:09:08.000809 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:08.000776 2565 patch_prober.go:28] interesting pod/image-registry-69c8bb978-vh2cc container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 02 19:09:08.001206 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:08.000832 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" podUID="964b1f60-6743-40ae-8ce3-3e69d1ed6181" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:09:08.456570 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:08.456533 2565 patch_prober.go:28] interesting pod/image-registry-544bbd9b4c-lw6hf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 02 19:09:08.456711 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:08.456597 2565 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" podUID="043ef48e-d5b2-4090-af93-39a4a503f8d1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:09:09.870621 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:09.870593 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-544bbd9b4c-lw6hf" Jul 02 19:09:11.430765 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.430725 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-slqtg"] Jul 02 19:09:11.444715 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.444683 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.447501 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.447469 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jgkx2\"" Jul 02 19:09:11.448520 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.448381 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 02 19:09:11.448520 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.448368 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 02 19:09:11.448520 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.448405 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 02 19:09:11.448520 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.448419 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 02 19:09:11.448520 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.448456 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 02 19:09:11.448520 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.448488 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 02 19:09:11.546222 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.546187 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-node-exporter-textfile\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.546583 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.546554 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-node-exporter-tls\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.547001 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.546967 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.547106 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.547009 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmswh\" (UniqueName: \"kubernetes.io/projected/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-kube-api-access-mmswh\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.547106 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.547044 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-sys\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.547106 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.547068 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-node-exporter-wtmp\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.547287 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.547153 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-root\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.547287 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.547188 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.547408 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.547287 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-metrics-client-ca\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.648736 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.648696 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-root\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.648736 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.648739 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.648988 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.648766 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-metrics-client-ca\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.648988 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.648795 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-node-exporter-textfile\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.648988 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.648820 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-node-exporter-tls\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.648988 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.648819 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-root\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.648988 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.648838 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.648988 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.648879 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmswh\" (UniqueName: \"kubernetes.io/projected/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-kube-api-access-mmswh\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.648988 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.648918 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-sys\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.648988 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.648943 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-node-exporter-wtmp\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.649381 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.649097 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-node-exporter-wtmp\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.649381 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.649217 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-node-exporter-textfile\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.649381 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.649282 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-sys\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.649541 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.649411 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.649589 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.649545 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-metrics-client-ca\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.651208 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.651186 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.651371 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.651335 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-node-exporter-tls\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.658976 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.658954 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmswh\" (UniqueName: \"kubernetes.io/projected/9e57abb6-b9be-42c1-9b2c-13d85e7700d8-kube-api-access-mmswh\") pod \"node-exporter-slqtg\" (UID: \"9e57abb6-b9be-42c1-9b2c-13d85e7700d8\") " pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.755040 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.754937 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-slqtg" Jul 02 19:09:11.784182 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:09:11.784147 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e57abb6_b9be_42c1_9b2c_13d85e7700d8.slice/crio-5f5371e69b501eba1b142601a6e2fbef45e5fea8ad173874d9f8930d6ce7bba7 WatchSource:0}: Error finding container 5f5371e69b501eba1b142601a6e2fbef45e5fea8ad173874d9f8930d6ce7bba7: Status 404 returned error can't find the container with id 5f5371e69b501eba1b142601a6e2fbef45e5fea8ad173874d9f8930d6ce7bba7 Jul 02 19:09:11.972256 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:11.972218 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slqtg" event={"ID":"9e57abb6-b9be-42c1-9b2c-13d85e7700d8","Type":"ContainerStarted","Data":"5f5371e69b501eba1b142601a6e2fbef45e5fea8ad173874d9f8930d6ce7bba7"} Jul 02 19:09:13.022176 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.022099 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" podUID="964b1f60-6743-40ae-8ce3-3e69d1ed6181" containerName="registry" containerID="cri-o://769a458f326adf35dfd9fc5d79d8f1db3d51790d64c35fb22ed32652a657a68e" gracePeriod=30 Jul 02 19:09:13.304266 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.304230 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:09:13.466332 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.466292 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-bound-sa-token\") pod \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " Jul 02 19:09:13.466516 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.466374 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/964b1f60-6743-40ae-8ce3-3e69d1ed6181-image-registry-private-configuration\") pod \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " Jul 02 19:09:13.466516 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.466422 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/964b1f60-6743-40ae-8ce3-3e69d1ed6181-ca-trust-extracted\") pod \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " Jul 02 19:09:13.466516 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.466440 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls\") pod \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " Jul 02 19:09:13.466516 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.466455 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q9fb\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-kube-api-access-8q9fb\") pod \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " Jul 02 19:09:13.466516 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.466475 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-certificates\") pod \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " Jul 02 19:09:13.466516 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.466505 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/964b1f60-6743-40ae-8ce3-3e69d1ed6181-installation-pull-secrets\") pod \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " Jul 02 19:09:13.466811 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.466537 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/964b1f60-6743-40ae-8ce3-3e69d1ed6181-trusted-ca\") pod \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\" (UID: \"964b1f60-6743-40ae-8ce3-3e69d1ed6181\") " Jul 02 19:09:13.467011 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.466976 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "964b1f60-6743-40ae-8ce3-3e69d1ed6181" (UID: "964b1f60-6743-40ae-8ce3-3e69d1ed6181"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:09:13.467124 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.467059 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/964b1f60-6743-40ae-8ce3-3e69d1ed6181-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "964b1f60-6743-40ae-8ce3-3e69d1ed6181" (UID: "964b1f60-6743-40ae-8ce3-3e69d1ed6181"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:09:13.475055 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.469369 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "964b1f60-6743-40ae-8ce3-3e69d1ed6181" (UID: "964b1f60-6743-40ae-8ce3-3e69d1ed6181"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:09:13.475055 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.474713 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-kube-api-access-8q9fb" (OuterVolumeSpecName: "kube-api-access-8q9fb") pod "964b1f60-6743-40ae-8ce3-3e69d1ed6181" (UID: "964b1f60-6743-40ae-8ce3-3e69d1ed6181"). InnerVolumeSpecName "kube-api-access-8q9fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:09:13.475597 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.475471 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/964b1f60-6743-40ae-8ce3-3e69d1ed6181-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "964b1f60-6743-40ae-8ce3-3e69d1ed6181" (UID: "964b1f60-6743-40ae-8ce3-3e69d1ed6181"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:09:13.475737 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.475598 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/964b1f60-6743-40ae-8ce3-3e69d1ed6181-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "964b1f60-6743-40ae-8ce3-3e69d1ed6181" (UID: "964b1f60-6743-40ae-8ce3-3e69d1ed6181"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:09:13.476051 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.476024 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "964b1f60-6743-40ae-8ce3-3e69d1ed6181" (UID: "964b1f60-6743-40ae-8ce3-3e69d1ed6181"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:09:13.479260 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.479225 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964b1f60-6743-40ae-8ce3-3e69d1ed6181-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "964b1f60-6743-40ae-8ce3-3e69d1ed6181" (UID: "964b1f60-6743-40ae-8ce3-3e69d1ed6181"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:09:13.525395 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.525339 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:09:13.529512 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.529453 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:09:13.567212 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.567176 2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/964b1f60-6743-40ae-8ce3-3e69d1ed6181-ca-trust-extracted\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:09:13.567212 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.567202 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-tls\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:09:13.567379 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.567220 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q9fb\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-kube-api-access-8q9fb\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:09:13.567379 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.567229 2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/964b1f60-6743-40ae-8ce3-3e69d1ed6181-registry-certificates\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:09:13.567379 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.567243 2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/964b1f60-6743-40ae-8ce3-3e69d1ed6181-installation-pull-secrets\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:09:13.567379 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.567257 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/964b1f60-6743-40ae-8ce3-3e69d1ed6181-trusted-ca\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:09:13.567379 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.567267 2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/964b1f60-6743-40ae-8ce3-3e69d1ed6181-bound-sa-token\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:09:13.567379 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.567276 2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/964b1f60-6743-40ae-8ce3-3e69d1ed6181-image-registry-private-configuration\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:09:13.981045 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.981004 2565 generic.go:356] "Generic (PLEG): container finished" podID="964b1f60-6743-40ae-8ce3-3e69d1ed6181" containerID="769a458f326adf35dfd9fc5d79d8f1db3d51790d64c35fb22ed32652a657a68e" exitCode=0 Jul 02 19:09:13.981293 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.981077 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" Jul 02 19:09:13.981293 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.981096 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" event={"ID":"964b1f60-6743-40ae-8ce3-3e69d1ed6181","Type":"ContainerDied","Data":"769a458f326adf35dfd9fc5d79d8f1db3d51790d64c35fb22ed32652a657a68e"} Jul 02 19:09:13.981293 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.981147 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69c8bb978-vh2cc" event={"ID":"964b1f60-6743-40ae-8ce3-3e69d1ed6181","Type":"ContainerDied","Data":"f1465d076b4b018a4c94a0c106e11e987e7f05bac4da5917aef5f5a6f1d6d027"} Jul 02 19:09:13.981293 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.981166 2565 scope.go:117] "RemoveContainer" containerID="769a458f326adf35dfd9fc5d79d8f1db3d51790d64c35fb22ed32652a657a68e" Jul 02 19:09:13.982770 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.982744 2565 generic.go:356] "Generic (PLEG): container finished" podID="9e57abb6-b9be-42c1-9b2c-13d85e7700d8" containerID="14202ade37413d456070358cbf930d38d148cdd55651e6b2958d203a88f79c87" exitCode=0 Jul 02 19:09:13.982871 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.982831 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slqtg" event={"ID":"9e57abb6-b9be-42c1-9b2c-13d85e7700d8","Type":"ContainerDied","Data":"14202ade37413d456070358cbf930d38d148cdd55651e6b2958d203a88f79c87"} Jul 02 19:09:13.989761 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.989741 2565 scope.go:117] "RemoveContainer" containerID="769a458f326adf35dfd9fc5d79d8f1db3d51790d64c35fb22ed32652a657a68e" Jul 02 19:09:13.990041 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:09:13.990018 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"769a458f326adf35dfd9fc5d79d8f1db3d51790d64c35fb22ed32652a657a68e\": container with ID starting with 769a458f326adf35dfd9fc5d79d8f1db3d51790d64c35fb22ed32652a657a68e not found: ID does not exist" containerID="769a458f326adf35dfd9fc5d79d8f1db3d51790d64c35fb22ed32652a657a68e" Jul 02 19:09:13.990084 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:13.990049 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769a458f326adf35dfd9fc5d79d8f1db3d51790d64c35fb22ed32652a657a68e"} err="failed to get container status \"769a458f326adf35dfd9fc5d79d8f1db3d51790d64c35fb22ed32652a657a68e\": rpc error: code = NotFound desc = could not find container \"769a458f326adf35dfd9fc5d79d8f1db3d51790d64c35fb22ed32652a657a68e\": container with ID starting with 769a458f326adf35dfd9fc5d79d8f1db3d51790d64c35fb22ed32652a657a68e not found: ID does not exist" Jul 02 19:09:14.021385 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:14.021360 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69c8bb978-vh2cc"] Jul 02 19:09:14.025842 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:14.025817 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69c8bb978-vh2cc"] Jul 02 19:09:14.989562 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:14.989531 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slqtg" event={"ID":"9e57abb6-b9be-42c1-9b2c-13d85e7700d8","Type":"ContainerStarted","Data":"6e0fdbc4868535ec5fbf86b326660c3e1025ebbe6f9fe772b2c986bcbcba91fd"} Jul 02 19:09:14.989736 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:14.989578 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slqtg" event={"ID":"9e57abb6-b9be-42c1-9b2c-13d85e7700d8","Type":"ContainerStarted","Data":"7cc96dbf6ea468cbe9e2bb25213f11c49262dade3053e408058ea6518ea04a3e"} Jul 02 19:09:15.013241 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:15.013179 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-slqtg" podStartSLOduration=2.245771895 podStartE2EDuration="4.013161644s" totalImagesPullingTime="1.660507713s" totalInitContainerRuntime="106.882036ms" isStatefulPod=true podCreationTimestamp="2026-07-02 19:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:09:15.012024977 +0000 UTC m=+94.127376218" watchObservedRunningTime="2026-07-02 19:09:15.013161644 +0000 UTC m=+94.128512867" Jul 02 19:09:15.399290 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:15.399195 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="964b1f60-6743-40ae-8ce3-3e69d1ed6181" path="/var/lib/kubelet/pods/964b1f60-6743-40ae-8ce3-3e69d1ed6181/volumes" Jul 02 19:09:17.850168 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:17.850141 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m66fc" Jul 02 19:09:27.320078 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:27.320038 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-844fb7b7bf-npslr"] Jul 02 19:09:44.095055 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:44.095021 2565 generic.go:356] "Generic (PLEG): container finished" podID="fa48a740-00cf-4f9c-815b-c96d531852de" containerID="031f3c33b7a211c7b6d4a0ac4141ea91d119668c6d423834f249bf4fb241ad0e" exitCode=0 Jul 02 19:09:44.095488 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:44.095085 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xlvvc" event={"ID":"fa48a740-00cf-4f9c-815b-c96d531852de","Type":"ContainerDied","Data":"031f3c33b7a211c7b6d4a0ac4141ea91d119668c6d423834f249bf4fb241ad0e"} Jul 02 19:09:44.095488 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:44.095435 2565 scope.go:117] "RemoveContainer" containerID="031f3c33b7a211c7b6d4a0ac4141ea91d119668c6d423834f249bf4fb241ad0e" Jul 02 19:09:45.100425 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:45.100393 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-xlvvc" event={"ID":"fa48a740-00cf-4f9c-815b-c96d531852de","Type":"ContainerStarted","Data":"dfdd38be46e35e9028660d521b4e8c18bbf4429152fe5445e01be3e1d3466c2d"} Jul 02 19:09:52.341634 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.341573 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-844fb7b7bf-npslr" podUID="cbc5a6aa-acdb-457b-aeca-59d5a20f09fb" containerName="console" containerID="cri-o://6c5a8e70618545ab23ddc34414a74b2c4bcabef830a38b0a5eb8a1446de0b351" gracePeriod=15 Jul 02 19:09:52.591779 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.591714 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:09:52.669581 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.669539 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-console-config\") pod \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " Jul 02 19:09:52.669581 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.669581 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-oauth-serving-cert\") pod \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " Jul 02 19:09:52.669819 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.669601 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-console-serving-cert\") pod \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " Jul 02 19:09:52.669819 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.669618 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vfwz\" (UniqueName: \"kubernetes.io/projected/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-kube-api-access-9vfwz\") pod \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " Jul 02 19:09:52.669819 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.669636 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-service-ca\") pod \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " Jul 02 19:09:52.669819 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.669660 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-console-oauth-config\") pod \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\" (UID: \"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb\") " Jul 02 19:09:52.670079 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.670034 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cbc5a6aa-acdb-457b-aeca-59d5a20f09fb" (UID: "cbc5a6aa-acdb-457b-aeca-59d5a20f09fb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:09:52.670079 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.670047 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-service-ca" (OuterVolumeSpecName: "service-ca") pod "cbc5a6aa-acdb-457b-aeca-59d5a20f09fb" (UID: "cbc5a6aa-acdb-457b-aeca-59d5a20f09fb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:09:52.670181 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.670118 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-console-config" (OuterVolumeSpecName: "console-config") pod "cbc5a6aa-acdb-457b-aeca-59d5a20f09fb" (UID: "cbc5a6aa-acdb-457b-aeca-59d5a20f09fb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:09:52.671944 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.671917 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cbc5a6aa-acdb-457b-aeca-59d5a20f09fb" (UID: "cbc5a6aa-acdb-457b-aeca-59d5a20f09fb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:09:52.672060 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.672039 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-kube-api-access-9vfwz" (OuterVolumeSpecName: "kube-api-access-9vfwz") pod "cbc5a6aa-acdb-457b-aeca-59d5a20f09fb" (UID: "cbc5a6aa-acdb-457b-aeca-59d5a20f09fb"). InnerVolumeSpecName "kube-api-access-9vfwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:09:52.672124 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.672046 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cbc5a6aa-acdb-457b-aeca-59d5a20f09fb" (UID: "cbc5a6aa-acdb-457b-aeca-59d5a20f09fb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:09:52.771068 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.771026 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-console-config\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:09:52.771068 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.771060 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-oauth-serving-cert\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:09:52.771068 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.771070 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-console-serving-cert\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:09:52.771068 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.771080 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vfwz\" (UniqueName: \"kubernetes.io/projected/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-kube-api-access-9vfwz\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:09:52.771341 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.771088 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-service-ca\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:09:52.771341 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:52.771096 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb-console-oauth-config\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:09:53.132098 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:53.132060 2565 generic.go:356] "Generic (PLEG): container finished" podID="cbc5a6aa-acdb-457b-aeca-59d5a20f09fb" containerID="6c5a8e70618545ab23ddc34414a74b2c4bcabef830a38b0a5eb8a1446de0b351" exitCode=0 Jul 02 19:09:53.132275 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:53.132106 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-844fb7b7bf-npslr" Jul 02 19:09:53.132275 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:53.132155 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-844fb7b7bf-npslr" event={"ID":"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb","Type":"ContainerDied","Data":"6c5a8e70618545ab23ddc34414a74b2c4bcabef830a38b0a5eb8a1446de0b351"} Jul 02 19:09:53.132275 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:53.132203 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-844fb7b7bf-npslr" event={"ID":"cbc5a6aa-acdb-457b-aeca-59d5a20f09fb","Type":"ContainerDied","Data":"1a19e35a6cb7a65c8616e7dbca9eb876ca72ff3bc741a9b0909fa29b16f51833"} Jul 02 19:09:53.132275 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:53.132229 2565 scope.go:117] "RemoveContainer" containerID="6c5a8e70618545ab23ddc34414a74b2c4bcabef830a38b0a5eb8a1446de0b351" Jul 02 19:09:53.144796 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:53.144769 2565 scope.go:117] "RemoveContainer" containerID="6c5a8e70618545ab23ddc34414a74b2c4bcabef830a38b0a5eb8a1446de0b351" Jul 02 19:09:53.145156 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:09:53.145133 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5a8e70618545ab23ddc34414a74b2c4bcabef830a38b0a5eb8a1446de0b351\": container with ID starting with 6c5a8e70618545ab23ddc34414a74b2c4bcabef830a38b0a5eb8a1446de0b351 not found: ID does not exist" containerID="6c5a8e70618545ab23ddc34414a74b2c4bcabef830a38b0a5eb8a1446de0b351" Jul 02 19:09:53.145216 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:53.145166 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5a8e70618545ab23ddc34414a74b2c4bcabef830a38b0a5eb8a1446de0b351"} err="failed to get container status \"6c5a8e70618545ab23ddc34414a74b2c4bcabef830a38b0a5eb8a1446de0b351\": rpc error: code = NotFound desc = could not find container \"6c5a8e70618545ab23ddc34414a74b2c4bcabef830a38b0a5eb8a1446de0b351\": container with ID starting with 6c5a8e70618545ab23ddc34414a74b2c4bcabef830a38b0a5eb8a1446de0b351 not found: ID does not exist" Jul 02 19:09:53.155651 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:53.155620 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-844fb7b7bf-npslr"] Jul 02 19:09:53.159927 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:53.159900 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-844fb7b7bf-npslr"] Jul 02 19:09:53.399059 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:53.398966 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbc5a6aa-acdb-457b-aeca-59d5a20f09fb" path="/var/lib/kubelet/pods/cbc5a6aa-acdb-457b-aeca-59d5a20f09fb/volumes" Jul 02 19:09:55.140946 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:55.140912 2565 generic.go:356] "Generic (PLEG): container finished" podID="da2642bb-e400-40ec-bded-5f145bbba018" containerID="3fe6b82918f8267d6b7cd5bebeb354e8ec95d65a7f032fedfe92cdd3b7d52b28" exitCode=0 Jul 02 19:09:55.141314 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:55.140985 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-gnf78" event={"ID":"da2642bb-e400-40ec-bded-5f145bbba018","Type":"ContainerDied","Data":"3fe6b82918f8267d6b7cd5bebeb354e8ec95d65a7f032fedfe92cdd3b7d52b28"} Jul 02 19:09:55.141314 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:55.141301 2565 scope.go:117] "RemoveContainer" containerID="3fe6b82918f8267d6b7cd5bebeb354e8ec95d65a7f032fedfe92cdd3b7d52b28" Jul 02 19:09:56.146318 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:09:56.146283 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-gnf78" event={"ID":"da2642bb-e400-40ec-bded-5f145bbba018","Type":"ContainerStarted","Data":"873522de7f8de42d53f4e32e8b51d9273a0717baac9c92d38a6fd55c5571535e"} Jul 02 19:10:39.403830 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.403044 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f54bb68d8-hpc59"] Jul 02 19:10:39.403830 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.403599 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="964b1f60-6743-40ae-8ce3-3e69d1ed6181" containerName="registry" Jul 02 19:10:39.403830 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.403625 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="964b1f60-6743-40ae-8ce3-3e69d1ed6181" containerName="registry" Jul 02 19:10:39.403830 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.403646 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc5a6aa-acdb-457b-aeca-59d5a20f09fb" containerName="console" Jul 02 19:10:39.403830 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.403662 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc5a6aa-acdb-457b-aeca-59d5a20f09fb" containerName="console" Jul 02 19:10:39.404591 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.403851 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="964b1f60-6743-40ae-8ce3-3e69d1ed6181" containerName="registry" Jul 02 19:10:39.404591 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.403876 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc5a6aa-acdb-457b-aeca-59d5a20f09fb" containerName="console" Jul 02 19:10:39.407564 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.407535 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.413320 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.413291 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 02 19:10:39.413604 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.413291 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 02 19:10:39.413688 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.413296 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 02 19:10:39.413688 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.413380 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 02 19:10:39.413795 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.413497 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 02 19:10:39.414408 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.414388 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 02 19:10:39.414711 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.414688 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 02 19:10:39.415815 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.415180 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kq4f6\"" Jul 02 19:10:39.415815 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.415283 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f54bb68d8-hpc59"] Jul 02 19:10:39.418974 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.418930 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 02 19:10:39.539571 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.539533 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43578419-7bbc-4aab-90df-4f7ae1b797db-console-oauth-config\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.539571 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.539584 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-service-ca\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.539795 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.539610 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-trusted-ca-bundle\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.539795 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.539630 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-oauth-serving-cert\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.539795 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.539650 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-console-config\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.539795 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.539700 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqvq5\" (UniqueName: \"kubernetes.io/projected/43578419-7bbc-4aab-90df-4f7ae1b797db-kube-api-access-gqvq5\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.539795 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.539745 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43578419-7bbc-4aab-90df-4f7ae1b797db-console-serving-cert\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.640758 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.640717 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43578419-7bbc-4aab-90df-4f7ae1b797db-console-oauth-config\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.640758 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.640766 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-service-ca\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.640992 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.640781 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-trusted-ca-bundle\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.640992 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.640796 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-oauth-serving-cert\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.640992 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.640824 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-console-config\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.640992 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.640850 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqvq5\" (UniqueName: \"kubernetes.io/projected/43578419-7bbc-4aab-90df-4f7ae1b797db-kube-api-access-gqvq5\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.640992 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.640887 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43578419-7bbc-4aab-90df-4f7ae1b797db-console-serving-cert\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.641696 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.641669 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-service-ca\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.641836 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.641722 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-oauth-serving-cert\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.641836 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.641722 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-console-config\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.642042 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.642017 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-trusted-ca-bundle\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.643422 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.643390 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43578419-7bbc-4aab-90df-4f7ae1b797db-console-oauth-config\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.643559 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.643542 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43578419-7bbc-4aab-90df-4f7ae1b797db-console-serving-cert\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.649483 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.649457 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqvq5\" (UniqueName: \"kubernetes.io/projected/43578419-7bbc-4aab-90df-4f7ae1b797db-kube-api-access-gqvq5\") pod \"console-5f54bb68d8-hpc59\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.719139 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.719029 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:39.859443 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:39.859414 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f54bb68d8-hpc59"] Jul 02 19:10:39.862006 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:10:39.861974 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43578419_7bbc_4aab_90df_4f7ae1b797db.slice/crio-3726ebc4d25d2d97cbf4337f0072f3f273e55a102ea85f3c4f88d99cafb99c79 WatchSource:0}: Error finding container 3726ebc4d25d2d97cbf4337f0072f3f273e55a102ea85f3c4f88d99cafb99c79: Status 404 returned error can't find the container with id 3726ebc4d25d2d97cbf4337f0072f3f273e55a102ea85f3c4f88d99cafb99c79 Jul 02 19:10:40.308133 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:40.308094 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f54bb68d8-hpc59" event={"ID":"43578419-7bbc-4aab-90df-4f7ae1b797db","Type":"ContainerStarted","Data":"96081332dd04f483e1db5438e728c0e0143a68740712b5d71a45423aaf59bd0d"} Jul 02 19:10:40.308315 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:40.308149 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f54bb68d8-hpc59" event={"ID":"43578419-7bbc-4aab-90df-4f7ae1b797db","Type":"ContainerStarted","Data":"3726ebc4d25d2d97cbf4337f0072f3f273e55a102ea85f3c4f88d99cafb99c79"} Jul 02 19:10:40.325510 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:40.325457 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f54bb68d8-hpc59" podStartSLOduration=1.325439239 podStartE2EDuration="1.325439239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:10:40.325160818 +0000 UTC m=+179.440512041" watchObservedRunningTime="2026-07-02 19:10:40.325439239 +0000 UTC m=+179.440790462" Jul 02 19:10:49.720297 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:49.720256 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:49.720821 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:49.720362 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:49.725149 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:49.725120 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:50.347276 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.347247 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:10:50.521451 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.521412 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-db55cf848-xclm5"] Jul 02 19:10:50.525082 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.525049 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.537510 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.537482 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-db55cf848-xclm5"] Jul 02 19:10:50.625275 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.625176 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-oauth-serving-cert\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.625275 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.625220 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6b2x\" (UniqueName: \"kubernetes.io/projected/61dbbe19-4442-4c68-9829-08e6603131a8-kube-api-access-b6b2x\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.625275 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.625253 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61dbbe19-4442-4c68-9829-08e6603131a8-console-oauth-config\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.625547 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.625283 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-service-ca\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.625547 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.625307 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-console-config\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.625547 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.625333 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-trusted-ca-bundle\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.625547 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.625421 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61dbbe19-4442-4c68-9829-08e6603131a8-console-serving-cert\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.726248 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.726207 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61dbbe19-4442-4c68-9829-08e6603131a8-console-serving-cert\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.726691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.726311 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-oauth-serving-cert\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.726691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.726382 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6b2x\" (UniqueName: \"kubernetes.io/projected/61dbbe19-4442-4c68-9829-08e6603131a8-kube-api-access-b6b2x\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.726691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.726428 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61dbbe19-4442-4c68-9829-08e6603131a8-console-oauth-config\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.726691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.726459 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-service-ca\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.726691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.726492 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-console-config\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.726691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.726533 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-trusted-ca-bundle\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.727184 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.727157 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-oauth-serving-cert\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.727262 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.727166 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-service-ca\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.727301 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.727275 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-console-config\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.727553 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.727536 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-trusted-ca-bundle\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.728753 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.728726 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61dbbe19-4442-4c68-9829-08e6603131a8-console-serving-cert\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.728932 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.728912 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61dbbe19-4442-4c68-9829-08e6603131a8-console-oauth-config\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.735386 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.735336 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6b2x\" (UniqueName: \"kubernetes.io/projected/61dbbe19-4442-4c68-9829-08e6603131a8-kube-api-access-b6b2x\") pod \"console-db55cf848-xclm5\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.835485 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.835438 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:10:50.971454 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:50.971419 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-db55cf848-xclm5"] Jul 02 19:10:50.974265 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:10:50.974238 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61dbbe19_4442_4c68_9829_08e6603131a8.slice/crio-0105cd507794a1a7dbb779fcb29889507b79ec3aafdc26aa01d091f79524b4ca WatchSource:0}: Error finding container 0105cd507794a1a7dbb779fcb29889507b79ec3aafdc26aa01d091f79524b4ca: Status 404 returned error can't find the container with id 0105cd507794a1a7dbb779fcb29889507b79ec3aafdc26aa01d091f79524b4ca Jul 02 19:10:51.348843 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:51.348802 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db55cf848-xclm5" event={"ID":"61dbbe19-4442-4c68-9829-08e6603131a8","Type":"ContainerStarted","Data":"d5237352dcea1efe79f260769cf2a9691414009f707a08e15ddfa8ec7c6b8721"} Jul 02 19:10:51.349020 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:51.348856 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db55cf848-xclm5" event={"ID":"61dbbe19-4442-4c68-9829-08e6603131a8","Type":"ContainerStarted","Data":"0105cd507794a1a7dbb779fcb29889507b79ec3aafdc26aa01d091f79524b4ca"} Jul 02 19:10:51.370810 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:10:51.370755 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-db55cf848-xclm5" podStartSLOduration=1.370737839 podStartE2EDuration="1.370737839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:10:51.369240876 +0000 UTC m=+190.484592136" watchObservedRunningTime="2026-07-02 19:10:51.370737839 +0000 UTC m=+190.486089061" Jul 02 19:11:00.836437 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:00.836384 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:11:00.836980 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:00.836458 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:11:00.841272 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:00.841247 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:11:01.387911 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:01.387882 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:11:01.461624 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:01.461586 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f54bb68d8-hpc59"] Jul 02 19:11:26.483671 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.483611 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5f54bb68d8-hpc59" podUID="43578419-7bbc-4aab-90df-4f7ae1b797db" containerName="console" containerID="cri-o://96081332dd04f483e1db5438e728c0e0143a68740712b5d71a45423aaf59bd0d" gracePeriod=15 Jul 02 19:11:26.727317 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.727292 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:11:26.817866 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.817778 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-console-config\") pod \"43578419-7bbc-4aab-90df-4f7ae1b797db\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " Jul 02 19:11:26.817866 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.817842 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-trusted-ca-bundle\") pod \"43578419-7bbc-4aab-90df-4f7ae1b797db\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " Jul 02 19:11:26.818091 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.817879 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-oauth-serving-cert\") pod \"43578419-7bbc-4aab-90df-4f7ae1b797db\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " Jul 02 19:11:26.818091 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.817900 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43578419-7bbc-4aab-90df-4f7ae1b797db-console-oauth-config\") pod \"43578419-7bbc-4aab-90df-4f7ae1b797db\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " Jul 02 19:11:26.818091 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.817931 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-service-ca\") pod \"43578419-7bbc-4aab-90df-4f7ae1b797db\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " Jul 02 19:11:26.818091 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.817966 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqvq5\" (UniqueName: \"kubernetes.io/projected/43578419-7bbc-4aab-90df-4f7ae1b797db-kube-api-access-gqvq5\") pod \"43578419-7bbc-4aab-90df-4f7ae1b797db\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " Jul 02 19:11:26.818091 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.817989 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43578419-7bbc-4aab-90df-4f7ae1b797db-console-serving-cert\") pod \"43578419-7bbc-4aab-90df-4f7ae1b797db\" (UID: \"43578419-7bbc-4aab-90df-4f7ae1b797db\") " Jul 02 19:11:26.818391 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.818328 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-console-config" (OuterVolumeSpecName: "console-config") pod "43578419-7bbc-4aab-90df-4f7ae1b797db" (UID: "43578419-7bbc-4aab-90df-4f7ae1b797db"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:11:26.818459 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.818337 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "43578419-7bbc-4aab-90df-4f7ae1b797db" (UID: "43578419-7bbc-4aab-90df-4f7ae1b797db"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:11:26.818459 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.818394 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "43578419-7bbc-4aab-90df-4f7ae1b797db" (UID: "43578419-7bbc-4aab-90df-4f7ae1b797db"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:11:26.818459 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.818423 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-service-ca" (OuterVolumeSpecName: "service-ca") pod "43578419-7bbc-4aab-90df-4f7ae1b797db" (UID: "43578419-7bbc-4aab-90df-4f7ae1b797db"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:11:26.820235 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.820207 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43578419-7bbc-4aab-90df-4f7ae1b797db-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "43578419-7bbc-4aab-90df-4f7ae1b797db" (UID: "43578419-7bbc-4aab-90df-4f7ae1b797db"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:11:26.820290 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.820241 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43578419-7bbc-4aab-90df-4f7ae1b797db-kube-api-access-gqvq5" (OuterVolumeSpecName: "kube-api-access-gqvq5") pod "43578419-7bbc-4aab-90df-4f7ae1b797db" (UID: "43578419-7bbc-4aab-90df-4f7ae1b797db"). InnerVolumeSpecName "kube-api-access-gqvq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:11:26.820290 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.820264 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43578419-7bbc-4aab-90df-4f7ae1b797db-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "43578419-7bbc-4aab-90df-4f7ae1b797db" (UID: "43578419-7bbc-4aab-90df-4f7ae1b797db"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:11:26.918879 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.918841 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43578419-7bbc-4aab-90df-4f7ae1b797db-console-oauth-config\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:11:26.918879 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.918875 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-service-ca\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:11:26.918879 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.918885 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqvq5\" (UniqueName: \"kubernetes.io/projected/43578419-7bbc-4aab-90df-4f7ae1b797db-kube-api-access-gqvq5\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:11:26.919113 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.918895 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43578419-7bbc-4aab-90df-4f7ae1b797db-console-serving-cert\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:11:26.919113 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.918905 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-console-config\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:11:26.919113 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.918913 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-trusted-ca-bundle\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:11:26.919113 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:26.918921 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43578419-7bbc-4aab-90df-4f7ae1b797db-oauth-serving-cert\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:11:27.481699 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:27.481663 2565 generic.go:356] "Generic (PLEG): container finished" podID="43578419-7bbc-4aab-90df-4f7ae1b797db" containerID="96081332dd04f483e1db5438e728c0e0143a68740712b5d71a45423aaf59bd0d" exitCode=0 Jul 02 19:11:27.481871 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:27.481714 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f54bb68d8-hpc59" Jul 02 19:11:27.481871 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:27.481736 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f54bb68d8-hpc59" event={"ID":"43578419-7bbc-4aab-90df-4f7ae1b797db","Type":"ContainerDied","Data":"96081332dd04f483e1db5438e728c0e0143a68740712b5d71a45423aaf59bd0d"} Jul 02 19:11:27.481871 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:27.481785 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f54bb68d8-hpc59" event={"ID":"43578419-7bbc-4aab-90df-4f7ae1b797db","Type":"ContainerDied","Data":"3726ebc4d25d2d97cbf4337f0072f3f273e55a102ea85f3c4f88d99cafb99c79"} Jul 02 19:11:27.481871 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:27.481815 2565 scope.go:117] "RemoveContainer" containerID="96081332dd04f483e1db5438e728c0e0143a68740712b5d71a45423aaf59bd0d" Jul 02 19:11:27.490174 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:27.489973 2565 scope.go:117] "RemoveContainer" containerID="96081332dd04f483e1db5438e728c0e0143a68740712b5d71a45423aaf59bd0d" Jul 02 19:11:27.490425 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:11:27.490253 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96081332dd04f483e1db5438e728c0e0143a68740712b5d71a45423aaf59bd0d\": container with ID starting with 96081332dd04f483e1db5438e728c0e0143a68740712b5d71a45423aaf59bd0d not found: ID does not exist" containerID="96081332dd04f483e1db5438e728c0e0143a68740712b5d71a45423aaf59bd0d" Jul 02 19:11:27.490425 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:27.490282 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96081332dd04f483e1db5438e728c0e0143a68740712b5d71a45423aaf59bd0d"} err="failed to get container status \"96081332dd04f483e1db5438e728c0e0143a68740712b5d71a45423aaf59bd0d\": rpc error: code = NotFound desc = could not find container \"96081332dd04f483e1db5438e728c0e0143a68740712b5d71a45423aaf59bd0d\": container with ID starting with 96081332dd04f483e1db5438e728c0e0143a68740712b5d71a45423aaf59bd0d not found: ID does not exist" Jul 02 19:11:27.497653 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:27.497619 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f54bb68d8-hpc59"] Jul 02 19:11:27.501692 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:27.501669 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f54bb68d8-hpc59"] Jul 02 19:11:29.398618 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:29.398579 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43578419-7bbc-4aab-90df-4f7ae1b797db" path="/var/lib/kubelet/pods/43578419-7bbc-4aab-90df-4f7ae1b797db/volumes" Jul 02 19:11:42.755630 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:42.755599 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77"] Jul 02 19:11:42.756018 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:42.755915 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43578419-7bbc-4aab-90df-4f7ae1b797db" containerName="console" Jul 02 19:11:42.756018 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:42.755927 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="43578419-7bbc-4aab-90df-4f7ae1b797db" containerName="console" Jul 02 19:11:42.756018 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:42.755997 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="43578419-7bbc-4aab-90df-4f7ae1b797db" containerName="console" Jul 02 19:11:42.758683 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:42.758665 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" Jul 02 19:11:42.761124 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:42.761102 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 02 19:11:42.761227 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:42.761175 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 02 19:11:42.762020 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:42.762001 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2sbhq\"" Jul 02 19:11:42.768438 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:42.768411 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77"] Jul 02 19:11:42.949620 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:42.949580 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f7f1784-f2af-4bff-bfd0-586ce15d7f84-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77\" (UID: \"3f7f1784-f2af-4bff-bfd0-586ce15d7f84\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" Jul 02 19:11:42.949826 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:42.949639 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f7f1784-f2af-4bff-bfd0-586ce15d7f84-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77\" (UID: \"3f7f1784-f2af-4bff-bfd0-586ce15d7f84\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" Jul 02 19:11:42.949826 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:42.949750 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grfdd\" (UniqueName: \"kubernetes.io/projected/3f7f1784-f2af-4bff-bfd0-586ce15d7f84-kube-api-access-grfdd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77\" (UID: \"3f7f1784-f2af-4bff-bfd0-586ce15d7f84\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" Jul 02 19:11:43.050870 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:43.050768 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f7f1784-f2af-4bff-bfd0-586ce15d7f84-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77\" (UID: \"3f7f1784-f2af-4bff-bfd0-586ce15d7f84\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" Jul 02 19:11:43.050870 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:43.050831 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grfdd\" (UniqueName: \"kubernetes.io/projected/3f7f1784-f2af-4bff-bfd0-586ce15d7f84-kube-api-access-grfdd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77\" (UID: \"3f7f1784-f2af-4bff-bfd0-586ce15d7f84\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" Jul 02 19:11:43.050870 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:43.050858 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f7f1784-f2af-4bff-bfd0-586ce15d7f84-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77\" (UID: \"3f7f1784-f2af-4bff-bfd0-586ce15d7f84\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" Jul 02 19:11:43.051251 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:43.051229 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f7f1784-f2af-4bff-bfd0-586ce15d7f84-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77\" (UID: \"3f7f1784-f2af-4bff-bfd0-586ce15d7f84\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" Jul 02 19:11:43.051251 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:43.051240 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f7f1784-f2af-4bff-bfd0-586ce15d7f84-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77\" (UID: \"3f7f1784-f2af-4bff-bfd0-586ce15d7f84\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" Jul 02 19:11:43.060470 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:43.060438 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grfdd\" (UniqueName: \"kubernetes.io/projected/3f7f1784-f2af-4bff-bfd0-586ce15d7f84-kube-api-access-grfdd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77\" (UID: \"3f7f1784-f2af-4bff-bfd0-586ce15d7f84\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" Jul 02 19:11:43.067186 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:43.067158 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" Jul 02 19:11:43.204668 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:43.204635 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77"] Jul 02 19:11:43.207464 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:11:43.207439 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7f1784_f2af_4bff_bfd0_586ce15d7f84.slice/crio-980e0fb9242423352b4d26a84a129781998260113ee7ce96bebdd195f24ff916 WatchSource:0}: Error finding container 980e0fb9242423352b4d26a84a129781998260113ee7ce96bebdd195f24ff916: Status 404 returned error can't find the container with id 980e0fb9242423352b4d26a84a129781998260113ee7ce96bebdd195f24ff916 Jul 02 19:11:43.542342 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:43.542303 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" event={"ID":"3f7f1784-f2af-4bff-bfd0-586ce15d7f84","Type":"ContainerStarted","Data":"980e0fb9242423352b4d26a84a129781998260113ee7ce96bebdd195f24ff916"} Jul 02 19:11:51.575297 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:51.575265 2565 generic.go:356] "Generic (PLEG): container finished" podID="3f7f1784-f2af-4bff-bfd0-586ce15d7f84" containerID="97087bd8698e4b945911c92786c8dd350998c34c47de60651245325ce6629bd9" exitCode=0 Jul 02 19:11:51.575698 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:51.575329 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" event={"ID":"3f7f1784-f2af-4bff-bfd0-586ce15d7f84","Type":"ContainerDied","Data":"97087bd8698e4b945911c92786c8dd350998c34c47de60651245325ce6629bd9"} Jul 02 19:11:54.589044 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:54.589006 2565 generic.go:356] "Generic (PLEG): container finished" podID="3f7f1784-f2af-4bff-bfd0-586ce15d7f84" containerID="fefafdda7d809baa903903cb934ed1fedccd8c8b08a3ac561ddd6a06f92f3575" exitCode=0 Jul 02 19:11:54.589433 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:11:54.589090 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" event={"ID":"3f7f1784-f2af-4bff-bfd0-586ce15d7f84","Type":"ContainerDied","Data":"fefafdda7d809baa903903cb934ed1fedccd8c8b08a3ac561ddd6a06f92f3575"} Jul 02 19:12:02.625636 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:02.625596 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" event={"ID":"3f7f1784-f2af-4bff-bfd0-586ce15d7f84","Type":"ContainerStarted","Data":"abcdd2bc21ee7bc72d1c79864c31db0b4f810586a1f578eed94881abf015ee95"} Jul 02 19:12:02.643232 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:02.643156 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" podStartSLOduration=3.089973292 podStartE2EDuration="20.643134004s" totalImagesPullingTime="17.553160712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:11:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:12:02.642509924 +0000 UTC m=+261.757861141" watchObservedRunningTime="2026-07-02 19:12:02.643134004 +0000 UTC m=+261.758485227" Jul 02 19:12:03.631492 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:03.631418 2565 generic.go:356] "Generic (PLEG): container finished" podID="3f7f1784-f2af-4bff-bfd0-586ce15d7f84" containerID="abcdd2bc21ee7bc72d1c79864c31db0b4f810586a1f578eed94881abf015ee95" exitCode=0 Jul 02 19:12:03.631492 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:03.631472 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" event={"ID":"3f7f1784-f2af-4bff-bfd0-586ce15d7f84","Type":"ContainerDied","Data":"abcdd2bc21ee7bc72d1c79864c31db0b4f810586a1f578eed94881abf015ee95"} Jul 02 19:12:04.805923 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:04.805882 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" Jul 02 19:12:04.930538 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:04.930502 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grfdd\" (UniqueName: \"kubernetes.io/projected/3f7f1784-f2af-4bff-bfd0-586ce15d7f84-kube-api-access-grfdd\") pod \"3f7f1784-f2af-4bff-bfd0-586ce15d7f84\" (UID: \"3f7f1784-f2af-4bff-bfd0-586ce15d7f84\") " Jul 02 19:12:04.930716 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:04.930569 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f7f1784-f2af-4bff-bfd0-586ce15d7f84-util\") pod \"3f7f1784-f2af-4bff-bfd0-586ce15d7f84\" (UID: \"3f7f1784-f2af-4bff-bfd0-586ce15d7f84\") " Jul 02 19:12:04.930716 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:04.930621 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f7f1784-f2af-4bff-bfd0-586ce15d7f84-bundle\") pod \"3f7f1784-f2af-4bff-bfd0-586ce15d7f84\" (UID: \"3f7f1784-f2af-4bff-bfd0-586ce15d7f84\") " Jul 02 19:12:04.931449 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:04.931414 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7f1784-f2af-4bff-bfd0-586ce15d7f84-bundle" (OuterVolumeSpecName: "bundle") pod "3f7f1784-f2af-4bff-bfd0-586ce15d7f84" (UID: "3f7f1784-f2af-4bff-bfd0-586ce15d7f84"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:12:04.932896 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:04.932868 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7f1784-f2af-4bff-bfd0-586ce15d7f84-kube-api-access-grfdd" (OuterVolumeSpecName: "kube-api-access-grfdd") pod "3f7f1784-f2af-4bff-bfd0-586ce15d7f84" (UID: "3f7f1784-f2af-4bff-bfd0-586ce15d7f84"). InnerVolumeSpecName "kube-api-access-grfdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:12:04.936099 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:04.936074 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7f1784-f2af-4bff-bfd0-586ce15d7f84-util" (OuterVolumeSpecName: "util") pod "3f7f1784-f2af-4bff-bfd0-586ce15d7f84" (UID: "3f7f1784-f2af-4bff-bfd0-586ce15d7f84"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:12:05.031953 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:05.031914 2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f7f1784-f2af-4bff-bfd0-586ce15d7f84-util\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:12:05.031953 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:05.031945 2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f7f1784-f2af-4bff-bfd0-586ce15d7f84-bundle\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:12:05.031953 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:05.031954 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grfdd\" (UniqueName: \"kubernetes.io/projected/3f7f1784-f2af-4bff-bfd0-586ce15d7f84-kube-api-access-grfdd\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:12:05.641394 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:05.641357 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" event={"ID":"3f7f1784-f2af-4bff-bfd0-586ce15d7f84","Type":"ContainerDied","Data":"980e0fb9242423352b4d26a84a129781998260113ee7ce96bebdd195f24ff916"} Jul 02 19:12:05.641394 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:05.641380 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jlw77" Jul 02 19:12:05.641590 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:05.641411 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="980e0fb9242423352b4d26a84a129781998260113ee7ce96bebdd195f24ff916" Jul 02 19:12:25.761222 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:25.761184 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-l4k6n"] Jul 02 19:12:25.761744 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:25.761585 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f7f1784-f2af-4bff-bfd0-586ce15d7f84" containerName="util" Jul 02 19:12:25.761744 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:25.761606 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f7f1784-f2af-4bff-bfd0-586ce15d7f84" containerName="util" Jul 02 19:12:25.761744 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:25.761620 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f7f1784-f2af-4bff-bfd0-586ce15d7f84" containerName="extract" Jul 02 19:12:25.761744 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:25.761628 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f7f1784-f2af-4bff-bfd0-586ce15d7f84" containerName="extract" Jul 02 19:12:25.761744 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:25.761651 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f7f1784-f2af-4bff-bfd0-586ce15d7f84" containerName="pull" Jul 02 19:12:25.761744 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:25.761659 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f7f1784-f2af-4bff-bfd0-586ce15d7f84" containerName="pull" Jul 02 19:12:25.761744 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:25.761732 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f7f1784-f2af-4bff-bfd0-586ce15d7f84" containerName="extract" Jul 02 19:12:25.763817 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:25.763797 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-l4k6n" Jul 02 19:12:25.766409 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:25.766387 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 02 19:12:25.766538 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:25.766387 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-2kl4f\"" Jul 02 19:12:25.766538 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:25.766416 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 02 19:12:25.772825 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:25.772800 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-l4k6n"] Jul 02 19:12:25.905190 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:25.905150 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b538f7b-eea3-425b-adfa-4e28d3e72837-bound-sa-token\") pod \"cert-manager-55b4d954d5-l4k6n\" (UID: \"8b538f7b-eea3-425b-adfa-4e28d3e72837\") " pod="cert-manager/cert-manager-55b4d954d5-l4k6n" Jul 02 19:12:25.905394 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:25.905221 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnr5c\" (UniqueName: \"kubernetes.io/projected/8b538f7b-eea3-425b-adfa-4e28d3e72837-kube-api-access-lnr5c\") pod \"cert-manager-55b4d954d5-l4k6n\" (UID: \"8b538f7b-eea3-425b-adfa-4e28d3e72837\") " pod="cert-manager/cert-manager-55b4d954d5-l4k6n" Jul 02 19:12:26.006520 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:26.006485 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b538f7b-eea3-425b-adfa-4e28d3e72837-bound-sa-token\") pod \"cert-manager-55b4d954d5-l4k6n\" (UID: \"8b538f7b-eea3-425b-adfa-4e28d3e72837\") " pod="cert-manager/cert-manager-55b4d954d5-l4k6n" Jul 02 19:12:26.006677 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:26.006553 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnr5c\" (UniqueName: \"kubernetes.io/projected/8b538f7b-eea3-425b-adfa-4e28d3e72837-kube-api-access-lnr5c\") pod \"cert-manager-55b4d954d5-l4k6n\" (UID: \"8b538f7b-eea3-425b-adfa-4e28d3e72837\") " pod="cert-manager/cert-manager-55b4d954d5-l4k6n" Jul 02 19:12:26.015186 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:26.015101 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b538f7b-eea3-425b-adfa-4e28d3e72837-bound-sa-token\") pod \"cert-manager-55b4d954d5-l4k6n\" (UID: \"8b538f7b-eea3-425b-adfa-4e28d3e72837\") " pod="cert-manager/cert-manager-55b4d954d5-l4k6n" Jul 02 19:12:26.015336 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:26.015289 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnr5c\" (UniqueName: \"kubernetes.io/projected/8b538f7b-eea3-425b-adfa-4e28d3e72837-kube-api-access-lnr5c\") pod \"cert-manager-55b4d954d5-l4k6n\" (UID: \"8b538f7b-eea3-425b-adfa-4e28d3e72837\") " pod="cert-manager/cert-manager-55b4d954d5-l4k6n" Jul 02 19:12:26.073392 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:26.073328 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-l4k6n" Jul 02 19:12:26.209598 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:26.209559 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-l4k6n"] Jul 02 19:12:26.212795 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:12:26.212763 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b538f7b_eea3_425b_adfa_4e28d3e72837.slice/crio-5ec81ece804492fba9a9bd5cb8b4e2e4b75f018457d44e13a5cca390c3720532 WatchSource:0}: Error finding container 5ec81ece804492fba9a9bd5cb8b4e2e4b75f018457d44e13a5cca390c3720532: Status 404 returned error can't find the container with id 5ec81ece804492fba9a9bd5cb8b4e2e4b75f018457d44e13a5cca390c3720532 Jul 02 19:12:26.718777 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:26.718742 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-l4k6n" event={"ID":"8b538f7b-eea3-425b-adfa-4e28d3e72837","Type":"ContainerStarted","Data":"5ec81ece804492fba9a9bd5cb8b4e2e4b75f018457d44e13a5cca390c3720532"} Jul 02 19:12:29.733541 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:29.733506 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-l4k6n" event={"ID":"8b538f7b-eea3-425b-adfa-4e28d3e72837","Type":"ContainerStarted","Data":"e63f5039c757721f832fbd2ef373b8e45d1a5c88479d7c5b804e44282f966da2"} Jul 02 19:12:29.754713 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:29.754658 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-l4k6n" podStartSLOduration=2.110012844 podStartE2EDuration="4.754637516s" totalImagesPullingTime="2.644624672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:12:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:12:29.753885782 +0000 UTC m=+288.869237006" watchObservedRunningTime="2026-07-02 19:12:29.754637516 +0000 UTC m=+288.869988739" Jul 02 19:12:41.293411 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:41.293378 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:12:41.301151 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:41.301124 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:12:41.304009 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:41.303986 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 02 19:12:43.708392 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:43.708360 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bwvhz"] Jul 02 19:12:43.710593 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:43.710578 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bwvhz" Jul 02 19:12:43.713825 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:43.713803 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:12:43.714749 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:43.714731 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-c242t\"" Jul 02 19:12:43.714811 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:43.714731 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 02 19:12:43.721753 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:43.721730 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bwvhz"] Jul 02 19:12:43.857732 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:43.857686 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck59l\" (UniqueName: \"kubernetes.io/projected/9536d1c3-68cd-4471-80dd-087cfb7b7391-kube-api-access-ck59l\") pod \"openshift-lws-operator-fd8ccff4c-bwvhz\" (UID: \"9536d1c3-68cd-4471-80dd-087cfb7b7391\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bwvhz" Jul 02 19:12:43.857732 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:43.857732 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9536d1c3-68cd-4471-80dd-087cfb7b7391-tmp\") pod \"openshift-lws-operator-fd8ccff4c-bwvhz\" (UID: \"9536d1c3-68cd-4471-80dd-087cfb7b7391\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bwvhz" Jul 02 19:12:43.958547 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:43.958461 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck59l\" (UniqueName: \"kubernetes.io/projected/9536d1c3-68cd-4471-80dd-087cfb7b7391-kube-api-access-ck59l\") pod \"openshift-lws-operator-fd8ccff4c-bwvhz\" (UID: \"9536d1c3-68cd-4471-80dd-087cfb7b7391\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bwvhz" Jul 02 19:12:43.958547 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:43.958503 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9536d1c3-68cd-4471-80dd-087cfb7b7391-tmp\") pod \"openshift-lws-operator-fd8ccff4c-bwvhz\" (UID: \"9536d1c3-68cd-4471-80dd-087cfb7b7391\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bwvhz" Jul 02 19:12:43.958861 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:43.958843 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9536d1c3-68cd-4471-80dd-087cfb7b7391-tmp\") pod \"openshift-lws-operator-fd8ccff4c-bwvhz\" (UID: \"9536d1c3-68cd-4471-80dd-087cfb7b7391\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bwvhz" Jul 02 19:12:43.967875 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:43.967834 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck59l\" (UniqueName: \"kubernetes.io/projected/9536d1c3-68cd-4471-80dd-087cfb7b7391-kube-api-access-ck59l\") pod \"openshift-lws-operator-fd8ccff4c-bwvhz\" (UID: \"9536d1c3-68cd-4471-80dd-087cfb7b7391\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bwvhz" Jul 02 19:12:44.019441 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:44.019402 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bwvhz" Jul 02 19:12:44.149945 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:44.149922 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bwvhz"] Jul 02 19:12:44.152407 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:12:44.152377 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9536d1c3_68cd_4471_80dd_087cfb7b7391.slice/crio-64d54a0c5f9e48ea4db0694e3f8d73ed3af3454b6416ad833626c6560057843d WatchSource:0}: Error finding container 64d54a0c5f9e48ea4db0694e3f8d73ed3af3454b6416ad833626c6560057843d: Status 404 returned error can't find the container with id 64d54a0c5f9e48ea4db0694e3f8d73ed3af3454b6416ad833626c6560057843d Jul 02 19:12:44.153973 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:44.153956 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:12:44.795413 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:44.795380 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bwvhz" event={"ID":"9536d1c3-68cd-4471-80dd-087cfb7b7391","Type":"ContainerStarted","Data":"64d54a0c5f9e48ea4db0694e3f8d73ed3af3454b6416ad833626c6560057843d"} Jul 02 19:12:47.810766 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:47.810726 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bwvhz" event={"ID":"9536d1c3-68cd-4471-80dd-087cfb7b7391","Type":"ContainerStarted","Data":"68ed2931a5fd3c46c8b0f4b92fd4f1aa7209730eaf4f8c0b4fb7ccf2c55b3284"} Jul 02 19:12:47.827056 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:12:47.827001 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-bwvhz" podStartSLOduration=1.925659305 podStartE2EDuration="4.826980118s" totalImagesPullingTime="2.901320813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:12:47.826030314 +0000 UTC m=+306.941381539" watchObservedRunningTime="2026-07-02 19:12:47.826980118 +0000 UTC m=+306.942331341" Jul 02 19:13:03.468109 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.468072 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx"] Jul 02 19:13:03.470823 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.470801 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" Jul 02 19:13:03.473984 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.473959 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 02 19:13:03.474154 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.474118 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 02 19:13:03.474284 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.474267 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zwcgm\"" Jul 02 19:13:03.476467 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.476409 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 02 19:13:03.476718 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.476702 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 02 19:13:03.491018 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.490972 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx"] Jul 02 19:13:03.500672 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.500641 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7ae933f-25af-4598-8fd8-4cb71a8ac1d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-cw5hx\" (UID: \"d7ae933f-25af-4598-8fd8-4cb71a8ac1d1\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" Jul 02 19:13:03.500849 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.500686 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7ae933f-25af-4598-8fd8-4cb71a8ac1d1-tmp\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-cw5hx\" (UID: \"d7ae933f-25af-4598-8fd8-4cb71a8ac1d1\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" Jul 02 19:13:03.500849 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.500723 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7ae933f-25af-4598-8fd8-4cb71a8ac1d1-manifests\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-cw5hx\" (UID: \"d7ae933f-25af-4598-8fd8-4cb71a8ac1d1\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" Jul 02 19:13:03.500849 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.500761 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f26p\" (UniqueName: \"kubernetes.io/projected/d7ae933f-25af-4598-8fd8-4cb71a8ac1d1-kube-api-access-7f26p\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-cw5hx\" (UID: \"d7ae933f-25af-4598-8fd8-4cb71a8ac1d1\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" Jul 02 19:13:03.500963 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.500886 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d7ae933f-25af-4598-8fd8-4cb71a8ac1d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-cw5hx\" (UID: \"d7ae933f-25af-4598-8fd8-4cb71a8ac1d1\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" Jul 02 19:13:03.601669 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.601615 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7ae933f-25af-4598-8fd8-4cb71a8ac1d1-tmp\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-cw5hx\" (UID: \"d7ae933f-25af-4598-8fd8-4cb71a8ac1d1\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" Jul 02 19:13:03.601865 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.601687 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7ae933f-25af-4598-8fd8-4cb71a8ac1d1-manifests\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-cw5hx\" (UID: \"d7ae933f-25af-4598-8fd8-4cb71a8ac1d1\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" Jul 02 19:13:03.601865 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.601705 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f26p\" (UniqueName: \"kubernetes.io/projected/d7ae933f-25af-4598-8fd8-4cb71a8ac1d1-kube-api-access-7f26p\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-cw5hx\" (UID: \"d7ae933f-25af-4598-8fd8-4cb71a8ac1d1\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" Jul 02 19:13:03.601865 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.601739 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d7ae933f-25af-4598-8fd8-4cb71a8ac1d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-cw5hx\" (UID: \"d7ae933f-25af-4598-8fd8-4cb71a8ac1d1\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" Jul 02 19:13:03.601865 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.601772 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7ae933f-25af-4598-8fd8-4cb71a8ac1d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-cw5hx\" (UID: \"d7ae933f-25af-4598-8fd8-4cb71a8ac1d1\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" Jul 02 19:13:03.602198 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.602167 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d7ae933f-25af-4598-8fd8-4cb71a8ac1d1-manifests\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-cw5hx\" (UID: \"d7ae933f-25af-4598-8fd8-4cb71a8ac1d1\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" Jul 02 19:13:03.604035 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.604009 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7ae933f-25af-4598-8fd8-4cb71a8ac1d1-tmp\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-cw5hx\" (UID: \"d7ae933f-25af-4598-8fd8-4cb71a8ac1d1\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" Jul 02 19:13:03.604134 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.604102 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d7ae933f-25af-4598-8fd8-4cb71a8ac1d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-cw5hx\" (UID: \"d7ae933f-25af-4598-8fd8-4cb71a8ac1d1\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" Jul 02 19:13:03.604185 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.604134 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7ae933f-25af-4598-8fd8-4cb71a8ac1d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-cw5hx\" (UID: \"d7ae933f-25af-4598-8fd8-4cb71a8ac1d1\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" Jul 02 19:13:03.612561 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.612534 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f26p\" (UniqueName: \"kubernetes.io/projected/d7ae933f-25af-4598-8fd8-4cb71a8ac1d1-kube-api-access-7f26p\") pod \"opendatahub-operator-controller-manager-d8ddf44b5-cw5hx\" (UID: \"d7ae933f-25af-4598-8fd8-4cb71a8ac1d1\") " pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" Jul 02 19:13:03.781699 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.781584 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" Jul 02 19:13:03.926512 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:03.926479 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx"] Jul 02 19:13:03.929286 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:13:03.929253 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ae933f_25af_4598_8fd8_4cb71a8ac1d1.slice/crio-bfb0722fca68add092895c11f1e9667fff492a0fcfe7913cdd9f4b231033003f WatchSource:0}: Error finding container bfb0722fca68add092895c11f1e9667fff492a0fcfe7913cdd9f4b231033003f: Status 404 returned error can't find the container with id bfb0722fca68add092895c11f1e9667fff492a0fcfe7913cdd9f4b231033003f Jul 02 19:13:04.880486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:04.880439 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" event={"ID":"d7ae933f-25af-4598-8fd8-4cb71a8ac1d1","Type":"ContainerStarted","Data":"bfb0722fca68add092895c11f1e9667fff492a0fcfe7913cdd9f4b231033003f"} Jul 02 19:13:07.897110 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:07.897070 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" event={"ID":"d7ae933f-25af-4598-8fd8-4cb71a8ac1d1","Type":"ContainerStarted","Data":"33f13ee92d5d797d2191b7ce5d52a6c7122f39d69c55545b0a4fde7d43c1269a"} Jul 02 19:13:08.903154 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:08.903119 2565 generic.go:356] "Generic (PLEG): container finished" podID="d7ae933f-25af-4598-8fd8-4cb71a8ac1d1" containerID="33f13ee92d5d797d2191b7ce5d52a6c7122f39d69c55545b0a4fde7d43c1269a" exitCode=0 Jul 02 19:13:08.903565 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:08.903179 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" event={"ID":"d7ae933f-25af-4598-8fd8-4cb71a8ac1d1","Type":"ContainerDied","Data":"33f13ee92d5d797d2191b7ce5d52a6c7122f39d69c55545b0a4fde7d43c1269a"} Jul 02 19:13:09.909153 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:09.909117 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" event={"ID":"d7ae933f-25af-4598-8fd8-4cb71a8ac1d1","Type":"ContainerStarted","Data":"2911a40cb1fbd6f4274f1f85d6cd0ae266c7d963c552b15f70d8e18c8b52a829"} Jul 02 19:13:09.909532 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:09.909248 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" Jul 02 19:13:09.931690 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:09.931634 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" podStartSLOduration=2.545041115 podStartE2EDuration="6.931615803s" totalImagesPullingTime="3.329473733s" totalInitContainerRuntime="1.057100955s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:13:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:13:09.929393391 +0000 UTC m=+329.044744631" watchObservedRunningTime="2026-07-02 19:13:09.931615803 +0000 UTC m=+329.046967082" Jul 02 19:13:20.915755 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:20.915716 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d8ddf44b5-cw5hx" Jul 02 19:13:26.077159 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.077126 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7bdf57ddb6-5k2qt"] Jul 02 19:13:26.084837 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.084810 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bdf57ddb6-5k2qt" Jul 02 19:13:26.087784 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.087755 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 02 19:13:26.087927 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.087809 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 02 19:13:26.087991 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.087973 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-44hnj\"" Jul 02 19:13:26.088059 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.088007 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 02 19:13:26.088059 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.088034 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 02 19:13:26.092538 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.092512 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bdf57ddb6-5k2qt"] Jul 02 19:13:26.202471 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.202439 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef604999-a8e7-41ac-88ef-c5465fe39c9d-tls-certs\") pod \"kube-auth-proxy-7bdf57ddb6-5k2qt\" (UID: \"ef604999-a8e7-41ac-88ef-c5465fe39c9d\") " pod="openshift-ingress/kube-auth-proxy-7bdf57ddb6-5k2qt" Jul 02 19:13:26.202670 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.202503 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef604999-a8e7-41ac-88ef-c5465fe39c9d-tmp\") pod \"kube-auth-proxy-7bdf57ddb6-5k2qt\" (UID: \"ef604999-a8e7-41ac-88ef-c5465fe39c9d\") " pod="openshift-ingress/kube-auth-proxy-7bdf57ddb6-5k2qt" Jul 02 19:13:26.202670 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.202562 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mm95\" (UniqueName: \"kubernetes.io/projected/ef604999-a8e7-41ac-88ef-c5465fe39c9d-kube-api-access-2mm95\") pod \"kube-auth-proxy-7bdf57ddb6-5k2qt\" (UID: \"ef604999-a8e7-41ac-88ef-c5465fe39c9d\") " pod="openshift-ingress/kube-auth-proxy-7bdf57ddb6-5k2qt" Jul 02 19:13:26.303748 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.303704 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef604999-a8e7-41ac-88ef-c5465fe39c9d-tls-certs\") pod \"kube-auth-proxy-7bdf57ddb6-5k2qt\" (UID: \"ef604999-a8e7-41ac-88ef-c5465fe39c9d\") " pod="openshift-ingress/kube-auth-proxy-7bdf57ddb6-5k2qt" Jul 02 19:13:26.303928 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.303759 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef604999-a8e7-41ac-88ef-c5465fe39c9d-tmp\") pod \"kube-auth-proxy-7bdf57ddb6-5k2qt\" (UID: \"ef604999-a8e7-41ac-88ef-c5465fe39c9d\") " pod="openshift-ingress/kube-auth-proxy-7bdf57ddb6-5k2qt" Jul 02 19:13:26.303928 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.303785 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mm95\" (UniqueName: \"kubernetes.io/projected/ef604999-a8e7-41ac-88ef-c5465fe39c9d-kube-api-access-2mm95\") pod \"kube-auth-proxy-7bdf57ddb6-5k2qt\" (UID: \"ef604999-a8e7-41ac-88ef-c5465fe39c9d\") " pod="openshift-ingress/kube-auth-proxy-7bdf57ddb6-5k2qt" Jul 02 19:13:26.306125 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.306086 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef604999-a8e7-41ac-88ef-c5465fe39c9d-tmp\") pod \"kube-auth-proxy-7bdf57ddb6-5k2qt\" (UID: \"ef604999-a8e7-41ac-88ef-c5465fe39c9d\") " pod="openshift-ingress/kube-auth-proxy-7bdf57ddb6-5k2qt" Jul 02 19:13:26.306387 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.306364 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef604999-a8e7-41ac-88ef-c5465fe39c9d-tls-certs\") pod \"kube-auth-proxy-7bdf57ddb6-5k2qt\" (UID: \"ef604999-a8e7-41ac-88ef-c5465fe39c9d\") " pod="openshift-ingress/kube-auth-proxy-7bdf57ddb6-5k2qt" Jul 02 19:13:26.313756 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.313730 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mm95\" (UniqueName: \"kubernetes.io/projected/ef604999-a8e7-41ac-88ef-c5465fe39c9d-kube-api-access-2mm95\") pod \"kube-auth-proxy-7bdf57ddb6-5k2qt\" (UID: \"ef604999-a8e7-41ac-88ef-c5465fe39c9d\") " pod="openshift-ingress/kube-auth-proxy-7bdf57ddb6-5k2qt" Jul 02 19:13:26.398879 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.398791 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bdf57ddb6-5k2qt" Jul 02 19:13:26.539230 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.537550 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bdf57ddb6-5k2qt"] Jul 02 19:13:26.985247 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:26.985210 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bdf57ddb6-5k2qt" event={"ID":"ef604999-a8e7-41ac-88ef-c5465fe39c9d","Type":"ContainerStarted","Data":"5aabb000951a675b2fff47c7790277895f94deda2042f91cc1a499faea75d9c3"} Jul 02 19:13:29.650927 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.650385 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5"] Jul 02 19:13:29.653557 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.653541 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5" Jul 02 19:13:29.656994 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.656968 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 02 19:13:29.656994 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.656993 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 02 19:13:29.657182 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.657024 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 02 19:13:29.657182 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.657027 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-g54kq\"" Jul 02 19:13:29.663397 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.663376 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5"] Jul 02 19:13:29.733620 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.733574 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a1a4e1ea-db6b-4c46-a318-090d19359a2f-manager-config\") pod \"lws-controller-manager-7d75bb4b7f-62xj5\" (UID: \"a1a4e1ea-db6b-4c46-a318-090d19359a2f\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5" Jul 02 19:13:29.733807 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.733689 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1a4e1ea-db6b-4c46-a318-090d19359a2f-cert\") pod \"lws-controller-manager-7d75bb4b7f-62xj5\" (UID: \"a1a4e1ea-db6b-4c46-a318-090d19359a2f\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5" Jul 02 19:13:29.733807 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.733741 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1a4e1ea-db6b-4c46-a318-090d19359a2f-metrics-cert\") pod \"lws-controller-manager-7d75bb4b7f-62xj5\" (UID: \"a1a4e1ea-db6b-4c46-a318-090d19359a2f\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5" Jul 02 19:13:29.733807 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.733781 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2lbk\" (UniqueName: \"kubernetes.io/projected/a1a4e1ea-db6b-4c46-a318-090d19359a2f-kube-api-access-j2lbk\") pod \"lws-controller-manager-7d75bb4b7f-62xj5\" (UID: \"a1a4e1ea-db6b-4c46-a318-090d19359a2f\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5" Jul 02 19:13:29.835118 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.835011 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1a4e1ea-db6b-4c46-a318-090d19359a2f-cert\") pod \"lws-controller-manager-7d75bb4b7f-62xj5\" (UID: \"a1a4e1ea-db6b-4c46-a318-090d19359a2f\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5" Jul 02 19:13:29.835118 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.835054 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1a4e1ea-db6b-4c46-a318-090d19359a2f-metrics-cert\") pod \"lws-controller-manager-7d75bb4b7f-62xj5\" (UID: \"a1a4e1ea-db6b-4c46-a318-090d19359a2f\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5" Jul 02 19:13:29.835118 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.835086 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2lbk\" (UniqueName: \"kubernetes.io/projected/a1a4e1ea-db6b-4c46-a318-090d19359a2f-kube-api-access-j2lbk\") pod \"lws-controller-manager-7d75bb4b7f-62xj5\" (UID: \"a1a4e1ea-db6b-4c46-a318-090d19359a2f\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5" Jul 02 19:13:29.835449 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.835129 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a1a4e1ea-db6b-4c46-a318-090d19359a2f-manager-config\") pod \"lws-controller-manager-7d75bb4b7f-62xj5\" (UID: \"a1a4e1ea-db6b-4c46-a318-090d19359a2f\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5" Jul 02 19:13:29.835761 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.835729 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a1a4e1ea-db6b-4c46-a318-090d19359a2f-manager-config\") pod \"lws-controller-manager-7d75bb4b7f-62xj5\" (UID: \"a1a4e1ea-db6b-4c46-a318-090d19359a2f\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5" Jul 02 19:13:29.837797 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.837772 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1a4e1ea-db6b-4c46-a318-090d19359a2f-cert\") pod \"lws-controller-manager-7d75bb4b7f-62xj5\" (UID: \"a1a4e1ea-db6b-4c46-a318-090d19359a2f\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5" Jul 02 19:13:29.837896 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.837772 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1a4e1ea-db6b-4c46-a318-090d19359a2f-metrics-cert\") pod \"lws-controller-manager-7d75bb4b7f-62xj5\" (UID: \"a1a4e1ea-db6b-4c46-a318-090d19359a2f\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5" Jul 02 19:13:29.852822 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.852794 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2lbk\" (UniqueName: \"kubernetes.io/projected/a1a4e1ea-db6b-4c46-a318-090d19359a2f-kube-api-access-j2lbk\") pod \"lws-controller-manager-7d75bb4b7f-62xj5\" (UID: \"a1a4e1ea-db6b-4c46-a318-090d19359a2f\") " pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5" Jul 02 19:13:29.964447 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:29.964412 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5" Jul 02 19:13:30.002482 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:30.002445 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bdf57ddb6-5k2qt" event={"ID":"ef604999-a8e7-41ac-88ef-c5465fe39c9d","Type":"ContainerStarted","Data":"816b2cc9d3c41e090a41da76b6daf69588aaa13e0abf12ba53120d32b6c140e1"} Jul 02 19:13:30.033189 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:30.033135 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7bdf57ddb6-5k2qt" podStartSLOduration=1.002718051 podStartE2EDuration="4.033116216s" totalImagesPullingTime="3.030398165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:13:30.032380859 +0000 UTC m=+349.147732109" watchObservedRunningTime="2026-07-02 19:13:30.033116216 +0000 UTC m=+349.148467440" Jul 02 19:13:30.115633 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:30.115608 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5"] Jul 02 19:13:30.118208 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:13:30.118177 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a4e1ea_db6b_4c46_a318_090d19359a2f.slice/crio-071c06e91e5ac500a06911c8f0ca2b46af308b8166ba79270780b08c5432f4a4 WatchSource:0}: Error finding container 071c06e91e5ac500a06911c8f0ca2b46af308b8166ba79270780b08c5432f4a4: Status 404 returned error can't find the container with id 071c06e91e5ac500a06911c8f0ca2b46af308b8166ba79270780b08c5432f4a4 Jul 02 19:13:31.007865 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:31.007825 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5" event={"ID":"a1a4e1ea-db6b-4c46-a318-090d19359a2f","Type":"ContainerStarted","Data":"071c06e91e5ac500a06911c8f0ca2b46af308b8166ba79270780b08c5432f4a4"} Jul 02 19:13:32.019905 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:32.019871 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5" event={"ID":"a1a4e1ea-db6b-4c46-a318-090d19359a2f","Type":"ContainerStarted","Data":"815144dd2e3d37dbceae1b849426b5b2335e4110d06e87293090871adf29f7d7"} Jul 02 19:13:32.020459 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:32.020017 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5" Jul 02 19:13:43.027272 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:43.027190 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5" Jul 02 19:13:43.049793 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:13:43.049727 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7d75bb4b7f-62xj5" podStartSLOduration=12.24249992 podStartE2EDuration="14.049705927s" totalImagesPullingTime="1.807206007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:13:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:13:32.042443214 +0000 UTC m=+351.157794455" watchObservedRunningTime="2026-07-02 19:13:43.049705927 +0000 UTC m=+362.165057150" Jul 02 19:15:55.013195 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:15:55.013157 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-db55cf848-xclm5"] Jul 02 19:16:20.040587 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.040522 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-db55cf848-xclm5" podUID="61dbbe19-4442-4c68-9829-08e6603131a8" containerName="console" containerID="cri-o://d5237352dcea1efe79f260769cf2a9691414009f707a08e15ddfa8ec7c6b8721" gracePeriod=15 Jul 02 19:16:20.285170 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.285137 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:16:20.381180 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.381086 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6b2x\" (UniqueName: \"kubernetes.io/projected/61dbbe19-4442-4c68-9829-08e6603131a8-kube-api-access-b6b2x\") pod \"61dbbe19-4442-4c68-9829-08e6603131a8\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " Jul 02 19:16:20.381180 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.381124 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61dbbe19-4442-4c68-9829-08e6603131a8-console-oauth-config\") pod \"61dbbe19-4442-4c68-9829-08e6603131a8\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " Jul 02 19:16:20.381180 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.381162 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-trusted-ca-bundle\") pod \"61dbbe19-4442-4c68-9829-08e6603131a8\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " Jul 02 19:16:20.381180 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.381180 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61dbbe19-4442-4c68-9829-08e6603131a8-console-serving-cert\") pod \"61dbbe19-4442-4c68-9829-08e6603131a8\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " Jul 02 19:16:20.381531 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.381238 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-console-config\") pod \"61dbbe19-4442-4c68-9829-08e6603131a8\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " Jul 02 19:16:20.381531 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.381284 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-oauth-serving-cert\") pod \"61dbbe19-4442-4c68-9829-08e6603131a8\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " Jul 02 19:16:20.381531 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.381322 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-service-ca\") pod \"61dbbe19-4442-4c68-9829-08e6603131a8\" (UID: \"61dbbe19-4442-4c68-9829-08e6603131a8\") " Jul 02 19:16:20.381811 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.381744 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-console-config" (OuterVolumeSpecName: "console-config") pod "61dbbe19-4442-4c68-9829-08e6603131a8" (UID: "61dbbe19-4442-4c68-9829-08e6603131a8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:16:20.381940 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.381811 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "61dbbe19-4442-4c68-9829-08e6603131a8" (UID: "61dbbe19-4442-4c68-9829-08e6603131a8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:16:20.381940 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.381819 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "61dbbe19-4442-4c68-9829-08e6603131a8" (UID: "61dbbe19-4442-4c68-9829-08e6603131a8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:16:20.381940 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.381903 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-service-ca" (OuterVolumeSpecName: "service-ca") pod "61dbbe19-4442-4c68-9829-08e6603131a8" (UID: "61dbbe19-4442-4c68-9829-08e6603131a8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:16:20.383496 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.383472 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61dbbe19-4442-4c68-9829-08e6603131a8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "61dbbe19-4442-4c68-9829-08e6603131a8" (UID: "61dbbe19-4442-4c68-9829-08e6603131a8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:16:20.383825 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.383807 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61dbbe19-4442-4c68-9829-08e6603131a8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "61dbbe19-4442-4c68-9829-08e6603131a8" (UID: "61dbbe19-4442-4c68-9829-08e6603131a8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:16:20.383892 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.383826 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61dbbe19-4442-4c68-9829-08e6603131a8-kube-api-access-b6b2x" (OuterVolumeSpecName: "kube-api-access-b6b2x") pod "61dbbe19-4442-4c68-9829-08e6603131a8" (UID: "61dbbe19-4442-4c68-9829-08e6603131a8"). InnerVolumeSpecName "kube-api-access-b6b2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:16:20.482306 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.482257 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6b2x\" (UniqueName: \"kubernetes.io/projected/61dbbe19-4442-4c68-9829-08e6603131a8-kube-api-access-b6b2x\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:16:20.482306 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.482300 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/61dbbe19-4442-4c68-9829-08e6603131a8-console-oauth-config\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:16:20.482306 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.482311 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-trusted-ca-bundle\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:16:20.482306 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.482321 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/61dbbe19-4442-4c68-9829-08e6603131a8-console-serving-cert\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:16:20.482601 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.482329 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-console-config\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:16:20.482601 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.482337 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-oauth-serving-cert\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:16:20.482601 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.482376 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/61dbbe19-4442-4c68-9829-08e6603131a8-service-ca\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:16:20.748581 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.748552 2565 generic.go:356] "Generic (PLEG): container finished" podID="61dbbe19-4442-4c68-9829-08e6603131a8" containerID="d5237352dcea1efe79f260769cf2a9691414009f707a08e15ddfa8ec7c6b8721" exitCode=0 Jul 02 19:16:20.748754 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.748598 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db55cf848-xclm5" Jul 02 19:16:20.748754 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.748608 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db55cf848-xclm5" event={"ID":"61dbbe19-4442-4c68-9829-08e6603131a8","Type":"ContainerDied","Data":"d5237352dcea1efe79f260769cf2a9691414009f707a08e15ddfa8ec7c6b8721"} Jul 02 19:16:20.748754 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.748644 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db55cf848-xclm5" event={"ID":"61dbbe19-4442-4c68-9829-08e6603131a8","Type":"ContainerDied","Data":"0105cd507794a1a7dbb779fcb29889507b79ec3aafdc26aa01d091f79524b4ca"} Jul 02 19:16:20.748754 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.748663 2565 scope.go:117] "RemoveContainer" containerID="d5237352dcea1efe79f260769cf2a9691414009f707a08e15ddfa8ec7c6b8721" Jul 02 19:16:20.757616 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.757598 2565 scope.go:117] "RemoveContainer" containerID="d5237352dcea1efe79f260769cf2a9691414009f707a08e15ddfa8ec7c6b8721" Jul 02 19:16:20.757863 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:16:20.757846 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5237352dcea1efe79f260769cf2a9691414009f707a08e15ddfa8ec7c6b8721\": container with ID starting with d5237352dcea1efe79f260769cf2a9691414009f707a08e15ddfa8ec7c6b8721 not found: ID does not exist" containerID="d5237352dcea1efe79f260769cf2a9691414009f707a08e15ddfa8ec7c6b8721" Jul 02 19:16:20.757913 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.757871 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5237352dcea1efe79f260769cf2a9691414009f707a08e15ddfa8ec7c6b8721"} err="failed to get container status \"d5237352dcea1efe79f260769cf2a9691414009f707a08e15ddfa8ec7c6b8721\": rpc error: code = NotFound desc = could not find container \"d5237352dcea1efe79f260769cf2a9691414009f707a08e15ddfa8ec7c6b8721\": container with ID starting with d5237352dcea1efe79f260769cf2a9691414009f707a08e15ddfa8ec7c6b8721 not found: ID does not exist" Jul 02 19:16:20.769819 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.769789 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-db55cf848-xclm5"] Jul 02 19:16:20.774572 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:20.774547 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-db55cf848-xclm5"] Jul 02 19:16:21.400182 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:21.400147 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61dbbe19-4442-4c68-9829-08e6603131a8" path="/var/lib/kubelet/pods/61dbbe19-4442-4c68-9829-08e6603131a8/volumes" Jul 02 19:16:25.891080 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:25.891048 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:16:25.891541 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:25.891363 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61dbbe19-4442-4c68-9829-08e6603131a8" containerName="console" Jul 02 19:16:25.891541 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:25.891378 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="61dbbe19-4442-4c68-9829-08e6603131a8" containerName="console" Jul 02 19:16:25.891541 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:25.891438 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="61dbbe19-4442-4c68-9829-08e6603131a8" containerName="console" Jul 02 19:16:25.896491 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:25.896474 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-s5gfq" Jul 02 19:16:25.898843 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:25.898823 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 02 19:16:25.898948 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:25.898848 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pfnst\"" Jul 02 19:16:25.899969 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:25.899947 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 02 19:16:25.900527 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:25.900381 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 02 19:16:25.902407 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:25.902386 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:16:25.927977 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:25.927942 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:16:26.032524 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:26.032488 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2a9bc9fe-c25f-40af-b54c-b9ff2d91b22b-config-file\") pod \"limitador-limitador-54d6d988d5-s5gfq\" (UID: \"2a9bc9fe-c25f-40af-b54c-b9ff2d91b22b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-s5gfq" Jul 02 19:16:26.032698 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:26.032569 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtndc\" (UniqueName: \"kubernetes.io/projected/2a9bc9fe-c25f-40af-b54c-b9ff2d91b22b-kube-api-access-dtndc\") pod \"limitador-limitador-54d6d988d5-s5gfq\" (UID: \"2a9bc9fe-c25f-40af-b54c-b9ff2d91b22b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-s5gfq" Jul 02 19:16:26.133652 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:26.133619 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtndc\" (UniqueName: \"kubernetes.io/projected/2a9bc9fe-c25f-40af-b54c-b9ff2d91b22b-kube-api-access-dtndc\") pod \"limitador-limitador-54d6d988d5-s5gfq\" (UID: \"2a9bc9fe-c25f-40af-b54c-b9ff2d91b22b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-s5gfq" Jul 02 19:16:26.133801 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:26.133671 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2a9bc9fe-c25f-40af-b54c-b9ff2d91b22b-config-file\") pod \"limitador-limitador-54d6d988d5-s5gfq\" (UID: \"2a9bc9fe-c25f-40af-b54c-b9ff2d91b22b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-s5gfq" Jul 02 19:16:26.134258 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:26.134241 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2a9bc9fe-c25f-40af-b54c-b9ff2d91b22b-config-file\") pod \"limitador-limitador-54d6d988d5-s5gfq\" (UID: \"2a9bc9fe-c25f-40af-b54c-b9ff2d91b22b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-s5gfq" Jul 02 19:16:26.142463 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:26.142393 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtndc\" (UniqueName: \"kubernetes.io/projected/2a9bc9fe-c25f-40af-b54c-b9ff2d91b22b-kube-api-access-dtndc\") pod \"limitador-limitador-54d6d988d5-s5gfq\" (UID: \"2a9bc9fe-c25f-40af-b54c-b9ff2d91b22b\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-s5gfq" Jul 02 19:16:26.206976 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:26.206938 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-s5gfq" Jul 02 19:16:26.343974 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:26.343947 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:16:26.346089 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:16:26.346043 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9bc9fe_c25f_40af_b54c_b9ff2d91b22b.slice/crio-dab030c9172c75e4d343d370ed1449d20bcdf5323f3e0d7f30131db5dead0239 WatchSource:0}: Error finding container dab030c9172c75e4d343d370ed1449d20bcdf5323f3e0d7f30131db5dead0239: Status 404 returned error can't find the container with id dab030c9172c75e4d343d370ed1449d20bcdf5323f3e0d7f30131db5dead0239 Jul 02 19:16:26.778583 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:26.778542 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-s5gfq" event={"ID":"2a9bc9fe-c25f-40af-b54c-b9ff2d91b22b","Type":"ContainerStarted","Data":"dab030c9172c75e4d343d370ed1449d20bcdf5323f3e0d7f30131db5dead0239"} Jul 02 19:16:29.791392 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:29.791335 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-s5gfq" event={"ID":"2a9bc9fe-c25f-40af-b54c-b9ff2d91b22b","Type":"ContainerStarted","Data":"7111aaa6754c6d710110c299f61ce7796b753cc0042f355e5cac5153c01a2bb8"} Jul 02 19:16:29.791774 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:29.791466 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-s5gfq" Jul 02 19:16:29.808337 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:29.808281 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-s5gfq" podStartSLOduration=2.414808539 podStartE2EDuration="4.808262938s" totalImagesPullingTime="2.393454399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:16:29.808036356 +0000 UTC m=+528.923387609" watchObservedRunningTime="2026-07-02 19:16:29.808262938 +0000 UTC m=+528.923614166" Jul 02 19:16:40.796310 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:16:40.796234 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-s5gfq" Jul 02 19:17:39.232478 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:17:39.232437 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:17:41.319650 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:17:41.319624 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:17:41.325231 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:17:41.325208 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:18:09.633766 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:18:09.633685 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:18:27.424151 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:18:27.424111 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:18:39.098002 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:18:39.097967 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:19:20.697250 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:20.697211 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:19:28.397136 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:28.397096 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:19:34.784800 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.784710 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s"] Jul 02 19:19:34.787976 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.787955 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:34.790746 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.790720 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 02 19:19:34.790927 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.790760 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 02 19:19:34.792134 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.792116 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 02 19:19:34.792246 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.792227 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mvwcw\"" Jul 02 19:19:34.796825 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.796791 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s"] Jul 02 19:19:34.869794 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.869747 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68722bde-8b53-4e6b-81e8-96044fb5de1d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s\" (UID: \"68722bde-8b53-4e6b-81e8-96044fb5de1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:34.869978 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.869848 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqll8\" (UniqueName: \"kubernetes.io/projected/68722bde-8b53-4e6b-81e8-96044fb5de1d-kube-api-access-cqll8\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s\" (UID: \"68722bde-8b53-4e6b-81e8-96044fb5de1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:34.869978 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.869915 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68722bde-8b53-4e6b-81e8-96044fb5de1d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s\" (UID: \"68722bde-8b53-4e6b-81e8-96044fb5de1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:34.869978 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.869939 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68722bde-8b53-4e6b-81e8-96044fb5de1d-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s\" (UID: \"68722bde-8b53-4e6b-81e8-96044fb5de1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:34.869978 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.869959 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68722bde-8b53-4e6b-81e8-96044fb5de1d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s\" (UID: \"68722bde-8b53-4e6b-81e8-96044fb5de1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:34.870117 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.870018 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68722bde-8b53-4e6b-81e8-96044fb5de1d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s\" (UID: \"68722bde-8b53-4e6b-81e8-96044fb5de1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:34.971056 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.971012 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqll8\" (UniqueName: \"kubernetes.io/projected/68722bde-8b53-4e6b-81e8-96044fb5de1d-kube-api-access-cqll8\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s\" (UID: \"68722bde-8b53-4e6b-81e8-96044fb5de1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:34.971242 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.971070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68722bde-8b53-4e6b-81e8-96044fb5de1d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s\" (UID: \"68722bde-8b53-4e6b-81e8-96044fb5de1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:34.971242 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.971193 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68722bde-8b53-4e6b-81e8-96044fb5de1d-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s\" (UID: \"68722bde-8b53-4e6b-81e8-96044fb5de1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:34.971242 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.971233 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68722bde-8b53-4e6b-81e8-96044fb5de1d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s\" (UID: \"68722bde-8b53-4e6b-81e8-96044fb5de1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:34.971438 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.971279 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68722bde-8b53-4e6b-81e8-96044fb5de1d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s\" (UID: \"68722bde-8b53-4e6b-81e8-96044fb5de1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:34.971438 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.971315 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68722bde-8b53-4e6b-81e8-96044fb5de1d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s\" (UID: \"68722bde-8b53-4e6b-81e8-96044fb5de1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:34.971547 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.971526 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68722bde-8b53-4e6b-81e8-96044fb5de1d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s\" (UID: \"68722bde-8b53-4e6b-81e8-96044fb5de1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:34.971629 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.971606 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68722bde-8b53-4e6b-81e8-96044fb5de1d-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s\" (UID: \"68722bde-8b53-4e6b-81e8-96044fb5de1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:34.971677 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.971648 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68722bde-8b53-4e6b-81e8-96044fb5de1d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s\" (UID: \"68722bde-8b53-4e6b-81e8-96044fb5de1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:34.973545 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.973522 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68722bde-8b53-4e6b-81e8-96044fb5de1d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s\" (UID: \"68722bde-8b53-4e6b-81e8-96044fb5de1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:34.973728 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.973711 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68722bde-8b53-4e6b-81e8-96044fb5de1d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s\" (UID: \"68722bde-8b53-4e6b-81e8-96044fb5de1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:34.979319 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:34.979290 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqll8\" (UniqueName: \"kubernetes.io/projected/68722bde-8b53-4e6b-81e8-96044fb5de1d-kube-api-access-cqll8\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s\" (UID: \"68722bde-8b53-4e6b-81e8-96044fb5de1d\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:35.098291 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:35.098188 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:35.234791 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:35.234764 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s"] Jul 02 19:19:35.237007 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:19:35.236974 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68722bde_8b53_4e6b_81e8_96044fb5de1d.slice/crio-f6ee983759b41b817f66bd01ac052bf97a6e7d132e9e8e79fc4b6547619f951a WatchSource:0}: Error finding container f6ee983759b41b817f66bd01ac052bf97a6e7d132e9e8e79fc4b6547619f951a: Status 404 returned error can't find the container with id f6ee983759b41b817f66bd01ac052bf97a6e7d132e9e8e79fc4b6547619f951a Jul 02 19:19:35.238408 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:35.238384 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:19:35.580258 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:35.580217 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" event={"ID":"68722bde-8b53-4e6b-81e8-96044fb5de1d","Type":"ContainerStarted","Data":"f6ee983759b41b817f66bd01ac052bf97a6e7d132e9e8e79fc4b6547619f951a"} Jul 02 19:19:36.586760 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:36.586671 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" event={"ID":"68722bde-8b53-4e6b-81e8-96044fb5de1d","Type":"ContainerStarted","Data":"de9730e61306c2383f100970d0207f4f186ae8a59331b670c3c6eb115e9495a6"} Jul 02 19:19:36.605225 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:36.605172 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" podStartSLOduration=1.585385821 podStartE2EDuration="2.605154525s" totalImagesPullingTime="1.019768704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:19:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:19:36.60377345 +0000 UTC m=+715.719124677" watchObservedRunningTime="2026-07-02 19:19:36.605154525 +0000 UTC m=+715.720505748" Jul 02 19:19:37.503362 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:37.503318 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:19:45.098598 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:45.098558 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:45.098986 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:45.098615 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:45.111782 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:45.111749 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:19:45.637116 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:19:45.637083 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s" Jul 02 19:20:56.741442 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:20:56.741405 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:21:09.390043 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:21:09.389964 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:21:18.388984 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:21:18.388947 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:21:27.197279 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:21:27.197241 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:21:37.989145 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:21:37.989111 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:22:05.896946 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:22:05.896910 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:22:14.992112 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:22:14.992077 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:22:41.348889 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:22:41.348806 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:22:41.352171 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:22:41.352145 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:22:47.293361 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:22:47.293305 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:22:55.984151 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:22:55.984109 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:23:05.998236 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:23:05.998199 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:23:11.386519 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:23:11.386479 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:23:22.491966 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:23:22.491929 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:23:31.596204 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:23:31.596160 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:23:42.911537 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:23:42.911501 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:23:50.197068 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:23:50.197031 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:23:57.993832 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:23:57.993797 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:24:06.307224 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:24:06.307143 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:24:18.601505 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:24:18.601465 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:24:30.286452 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:24:30.286415 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:24:54.404587 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:24:54.404549 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:25:05.193055 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:25:05.193014 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:25:21.043649 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:25:21.043609 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:25:30.188073 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:25:30.187980 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:25:39.889840 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:25:39.889760 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:25:51.389778 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:25:51.389741 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:26:00.395517 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:26:00.395481 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:27:41.376219 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:27:41.376188 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:27:41.376766 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:27:41.376499 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:27:47.195141 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:27:47.195103 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:27:57.298500 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:27:57.298463 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:28:08.108706 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:28:08.108667 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:28:20.397441 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:28:20.397408 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:28:32.408147 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:28:32.408111 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:28:41.511652 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:28:41.511571 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:28:51.594573 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:28:51.594537 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:29:02.032606 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:29:02.032560 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:29:12.521174 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:29:12.521139 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:29:22.616945 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:29:22.616907 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:29:33.205694 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:29:33.205652 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:29:54.807901 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:29:54.807863 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:30:04.196750 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:30:04.196660 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:30:14.296642 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:30:14.296593 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:30:24.892495 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:30:24.892456 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:30:42.195364 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:30:42.195311 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:30:52.598455 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:30:52.598404 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:31:01.903968 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:31:01.903930 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:31:11.786420 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:31:11.786377 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:31:22.216663 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:31:22.216628 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:31:31.391147 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:31:31.391102 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:31:44.691317 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:31:44.691227 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:31:54.198082 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:31:54.198043 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:32:03.495568 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:32:03.495527 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:32:14.194518 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:32:14.194480 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:32:24.994853 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:32:24.994815 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:32:31.493563 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:32:31.493527 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:32:41.401553 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:32:41.401526 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:32:41.402473 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:32:41.402450 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:32:47.093646 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:32:47.093603 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:32:58.300133 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:32:58.300096 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:33:08.791451 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:33:08.791369 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:33:21.603134 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:33:21.603077 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:33:30.289914 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:33:30.289874 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:33:41.193691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:33:41.193648 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:33:58.294304 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:33:58.294265 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:34:09.593338 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:34:09.593299 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:35:48.897508 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:35:48.897470 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:35:56.009945 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:35:56.009905 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:36:03.400097 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:36:03.400065 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:36:11.291990 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:36:11.291945 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:36:24.802035 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:36:24.801996 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:36:54.301740 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:36:54.301703 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:37:13.051877 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.051838 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6"] Jul 02 19:37:13.055092 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.055075 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.058828 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.058793 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a603bec7\"/\"kube-root-ca.crt\"" Jul 02 19:37:13.059416 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.059394 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a603bec7\"/\"e2e-lifecycle-model-a603bec7-kserve-self-signed-certs\"" Jul 02 19:37:13.059812 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.059794 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-a603bec7\"/\"default-dockercfg-ttqvb\"" Jul 02 19:37:13.059910 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.059843 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-a603bec7\"/\"openshift-service-ca.crt\"" Jul 02 19:37:13.078879 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.078780 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6"] Jul 02 19:37:13.112323 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.112288 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-dshm\") pod \"e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.112516 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.112332 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-model-cache\") pod \"e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.112516 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.112377 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-tmp-dir\") pod \"e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.112516 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.112458 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df7009f-3b3b-4484-b6bc-8136c23ba506-tls-certs\") pod \"e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.112516 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.112500 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7mh9\" (UniqueName: \"kubernetes.io/projected/9df7009f-3b3b-4484-b6bc-8136c23ba506-kube-api-access-k7mh9\") pod \"e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.112652 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.112542 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-home\") pod \"e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.213108 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.213050 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7mh9\" (UniqueName: \"kubernetes.io/projected/9df7009f-3b3b-4484-b6bc-8136c23ba506-kube-api-access-k7mh9\") pod \"e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.213108 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.213111 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-home\") pod \"e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.213400 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.213140 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-dshm\") pod \"e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.213400 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.213168 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-model-cache\") pod \"e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.213400 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.213192 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-tmp-dir\") pod \"e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.213400 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.213226 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df7009f-3b3b-4484-b6bc-8136c23ba506-tls-certs\") pod \"e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.213663 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.213640 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-model-cache\") pod \"e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.213717 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.213691 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-tmp-dir\") pod \"e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.213763 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.213740 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-home\") pod \"e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.215470 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.215451 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-dshm\") pod \"e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.215742 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.215724 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df7009f-3b3b-4484-b6bc-8136c23ba506-tls-certs\") pod \"e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.223252 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.223227 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7mh9\" (UniqueName: \"kubernetes.io/projected/9df7009f-3b3b-4484-b6bc-8136c23ba506-kube-api-access-k7mh9\") pod \"e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.365660 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.365556 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:13.510241 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:13.510214 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6"] Jul 02 19:37:13.511996 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:37:13.511967 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9df7009f_3b3b_4484_b6bc_8136c23ba506.slice/crio-2f25da3807e0fbd0d19c39da9ada20a1e325ff07016ed0c27478280f88d3b787 WatchSource:0}: Error finding container 2f25da3807e0fbd0d19c39da9ada20a1e325ff07016ed0c27478280f88d3b787: Status 404 returned error can't find the container with id 2f25da3807e0fbd0d19c39da9ada20a1e325ff07016ed0c27478280f88d3b787 Jul 02 19:37:14.114679 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:14.114646 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" event={"ID":"9df7009f-3b3b-4484-b6bc-8136c23ba506","Type":"ContainerStarted","Data":"18e93362e2cb1487d87612a99207ace5b0a437717c31ee2b7ab6aa7d040c2f79"} Jul 02 19:37:14.115065 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:14.114688 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" event={"ID":"9df7009f-3b3b-4484-b6bc-8136c23ba506","Type":"ContainerStarted","Data":"2f25da3807e0fbd0d19c39da9ada20a1e325ff07016ed0c27478280f88d3b787"} Jul 02 19:37:14.138447 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:14.138393 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" podStartSLOduration=1.138378079 podStartE2EDuration="1.138378079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:37:14.137598243 +0000 UTC m=+1773.252949467" watchObservedRunningTime="2026-07-02 19:37:14.138378079 +0000 UTC m=+1773.253729302" Jul 02 19:37:16.691073 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:16.691039 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:37:23.366420 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:23.366373 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:23.366420 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:23.366424 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:23.378827 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:23.378796 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:24.171694 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:24.171664 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:28.205127 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:28.205090 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:37:41.424765 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:41.424733 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:37:41.427258 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:41.427236 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:37:41.988302 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:41.988266 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6"] Jul 02 19:37:42.195317 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:42.195279 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:37:57.008452 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.008405 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" podUID="9df7009f-3b3b-4484-b6bc-8136c23ba506" containerName="main" containerID="cri-o://18e93362e2cb1487d87612a99207ace5b0a437717c31ee2b7ab6aa7d040c2f79" gracePeriod=45 Jul 02 19:37:57.259750 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.259689 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:57.271564 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.271541 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df7009f-3b3b-4484-b6bc-8136c23ba506-tls-certs\") pod \"9df7009f-3b3b-4484-b6bc-8136c23ba506\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " Jul 02 19:37:57.271646 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.271590 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-dshm\") pod \"9df7009f-3b3b-4484-b6bc-8136c23ba506\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " Jul 02 19:37:57.271646 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.271613 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-home\") pod \"9df7009f-3b3b-4484-b6bc-8136c23ba506\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " Jul 02 19:37:57.271738 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.271645 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-model-cache\") pod \"9df7009f-3b3b-4484-b6bc-8136c23ba506\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " Jul 02 19:37:57.271738 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.271676 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7mh9\" (UniqueName: \"kubernetes.io/projected/9df7009f-3b3b-4484-b6bc-8136c23ba506-kube-api-access-k7mh9\") pod \"9df7009f-3b3b-4484-b6bc-8136c23ba506\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " Jul 02 19:37:57.271738 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.271702 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-tmp-dir\") pod \"9df7009f-3b3b-4484-b6bc-8136c23ba506\" (UID: \"9df7009f-3b3b-4484-b6bc-8136c23ba506\") " Jul 02 19:37:57.271976 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.271938 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-model-cache" (OuterVolumeSpecName: "model-cache") pod "9df7009f-3b3b-4484-b6bc-8136c23ba506" (UID: "9df7009f-3b3b-4484-b6bc-8136c23ba506"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:37:57.271976 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.271949 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-home" (OuterVolumeSpecName: "home") pod "9df7009f-3b3b-4484-b6bc-8136c23ba506" (UID: "9df7009f-3b3b-4484-b6bc-8136c23ba506"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:37:57.272164 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.272042 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9df7009f-3b3b-4484-b6bc-8136c23ba506" (UID: "9df7009f-3b3b-4484-b6bc-8136c23ba506"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:37:57.273887 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.273863 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-dshm" (OuterVolumeSpecName: "dshm") pod "9df7009f-3b3b-4484-b6bc-8136c23ba506" (UID: "9df7009f-3b3b-4484-b6bc-8136c23ba506"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:37:57.273984 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.273877 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df7009f-3b3b-4484-b6bc-8136c23ba506-kube-api-access-k7mh9" (OuterVolumeSpecName: "kube-api-access-k7mh9") pod "9df7009f-3b3b-4484-b6bc-8136c23ba506" (UID: "9df7009f-3b3b-4484-b6bc-8136c23ba506"). InnerVolumeSpecName "kube-api-access-k7mh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:37:57.273984 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.273912 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df7009f-3b3b-4484-b6bc-8136c23ba506-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9df7009f-3b3b-4484-b6bc-8136c23ba506" (UID: "9df7009f-3b3b-4484-b6bc-8136c23ba506"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:37:57.304863 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.304832 2565 generic.go:356] "Generic (PLEG): container finished" podID="9df7009f-3b3b-4484-b6bc-8136c23ba506" containerID="18e93362e2cb1487d87612a99207ace5b0a437717c31ee2b7ab6aa7d040c2f79" exitCode=0 Jul 02 19:37:57.305048 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.304884 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" Jul 02 19:37:57.305048 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.304916 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" event={"ID":"9df7009f-3b3b-4484-b6bc-8136c23ba506","Type":"ContainerDied","Data":"18e93362e2cb1487d87612a99207ace5b0a437717c31ee2b7ab6aa7d040c2f79"} Jul 02 19:37:57.305048 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.304967 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6" event={"ID":"9df7009f-3b3b-4484-b6bc-8136c23ba506","Type":"ContainerDied","Data":"2f25da3807e0fbd0d19c39da9ada20a1e325ff07016ed0c27478280f88d3b787"} Jul 02 19:37:57.305048 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.304991 2565 scope.go:117] "RemoveContainer" containerID="18e93362e2cb1487d87612a99207ace5b0a437717c31ee2b7ab6aa7d040c2f79" Jul 02 19:37:57.314452 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.314335 2565 scope.go:117] "RemoveContainer" containerID="18e93362e2cb1487d87612a99207ace5b0a437717c31ee2b7ab6aa7d040c2f79" Jul 02 19:37:57.314737 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:37:57.314717 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e93362e2cb1487d87612a99207ace5b0a437717c31ee2b7ab6aa7d040c2f79\": container with ID starting with 18e93362e2cb1487d87612a99207ace5b0a437717c31ee2b7ab6aa7d040c2f79 not found: ID does not exist" containerID="18e93362e2cb1487d87612a99207ace5b0a437717c31ee2b7ab6aa7d040c2f79" Jul 02 19:37:57.314785 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.314747 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e93362e2cb1487d87612a99207ace5b0a437717c31ee2b7ab6aa7d040c2f79"} err="failed to get container status \"18e93362e2cb1487d87612a99207ace5b0a437717c31ee2b7ab6aa7d040c2f79\": rpc error: code = NotFound desc = could not find container \"18e93362e2cb1487d87612a99207ace5b0a437717c31ee2b7ab6aa7d040c2f79\": container with ID starting with 18e93362e2cb1487d87612a99207ace5b0a437717c31ee2b7ab6aa7d040c2f79 not found: ID does not exist" Jul 02 19:37:57.326835 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.326805 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6"] Jul 02 19:37:57.330395 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.330373 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-a603bec7/e2e-lifecycle-model-a603bec7-kserve-6cdb958575-krtb6"] Jul 02 19:37:57.373155 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.373127 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df7009f-3b3b-4484-b6bc-8136c23ba506-tls-certs\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:37:57.373155 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.373152 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-dshm\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:37:57.373155 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.373160 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-home\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:37:57.373385 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.373169 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-model-cache\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:37:57.373385 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.373176 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7mh9\" (UniqueName: \"kubernetes.io/projected/9df7009f-3b3b-4484-b6bc-8136c23ba506-kube-api-access-k7mh9\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:37:57.373385 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.373207 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df7009f-3b3b-4484-b6bc-8136c23ba506-tmp-dir\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:37:57.399912 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:37:57.399883 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9df7009f-3b3b-4484-b6bc-8136c23ba506" path="/var/lib/kubelet/pods/9df7009f-3b3b-4484-b6bc-8136c23ba506/volumes" Jul 02 19:38:21.590802 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:38:21.590763 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:39:30.698603 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:39:30.698550 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:39:45.188731 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:39:45.188694 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:40:00.006770 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.006732 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh"] Jul 02 19:40:00.007151 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.007029 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9df7009f-3b3b-4484-b6bc-8136c23ba506" containerName="main" Jul 02 19:40:00.007151 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.007039 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9df7009f-3b3b-4484-b6bc-8136c23ba506" containerName="main" Jul 02 19:40:00.007151 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.007093 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9df7009f-3b3b-4484-b6bc-8136c23ba506" containerName="main" Jul 02 19:40:00.009982 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.009964 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.012750 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.012724 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-09f1f6\"/\"openshift-service-ca.crt\"" Jul 02 19:40:00.012750 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.012724 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-09f1f6\"/\"test-model-09f1f6-kserve-self-signed-certs\"" Jul 02 19:40:00.013705 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.013682 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-09f1f6\"/\"kube-root-ca.crt\"" Jul 02 19:40:00.013813 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.013701 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-09f1f6\"/\"default-dockercfg-7kbcj\"" Jul 02 19:40:00.019226 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.019199 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh"] Jul 02 19:40:00.114472 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.114431 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-tmp-dir\") pod \"test-model-09f1f6-kserve-5596df4bfc-p55zh\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.114472 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.114474 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-dshm\") pod \"test-model-09f1f6-kserve-5596df4bfc-p55zh\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.114691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.114510 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-home\") pod \"test-model-09f1f6-kserve-5596df4bfc-p55zh\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.114691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.114560 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rn52\" (UniqueName: \"kubernetes.io/projected/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-kube-api-access-2rn52\") pod \"test-model-09f1f6-kserve-5596df4bfc-p55zh\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.114691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.114636 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-tls-certs\") pod \"test-model-09f1f6-kserve-5596df4bfc-p55zh\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.114691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.114671 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-model-cache\") pod \"test-model-09f1f6-kserve-5596df4bfc-p55zh\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.215238 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.215200 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-tls-certs\") pod \"test-model-09f1f6-kserve-5596df4bfc-p55zh\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.215420 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.215241 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-model-cache\") pod \"test-model-09f1f6-kserve-5596df4bfc-p55zh\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.215420 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.215279 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-tmp-dir\") pod \"test-model-09f1f6-kserve-5596df4bfc-p55zh\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.215420 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.215307 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-dshm\") pod \"test-model-09f1f6-kserve-5596df4bfc-p55zh\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.215420 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.215339 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-home\") pod \"test-model-09f1f6-kserve-5596df4bfc-p55zh\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.215615 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.215417 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rn52\" (UniqueName: \"kubernetes.io/projected/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-kube-api-access-2rn52\") pod \"test-model-09f1f6-kserve-5596df4bfc-p55zh\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.215834 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.215798 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-tmp-dir\") pod \"test-model-09f1f6-kserve-5596df4bfc-p55zh\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.215946 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.215842 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-model-cache\") pod \"test-model-09f1f6-kserve-5596df4bfc-p55zh\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.215946 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.215881 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-home\") pod \"test-model-09f1f6-kserve-5596df4bfc-p55zh\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.217660 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.217632 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-dshm\") pod \"test-model-09f1f6-kserve-5596df4bfc-p55zh\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.217915 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.217896 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-tls-certs\") pod \"test-model-09f1f6-kserve-5596df4bfc-p55zh\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.224023 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.223998 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rn52\" (UniqueName: \"kubernetes.io/projected/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-kube-api-access-2rn52\") pod \"test-model-09f1f6-kserve-5596df4bfc-p55zh\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.319462 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.319341 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:00.457727 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.457701 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh"] Jul 02 19:40:00.460109 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:40:00.460082 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf9ca157_75e5_4fab_886e_5e0ce7eafb7d.slice/crio-1b0921cee64c7b8069550c57ef472d7b66fd71d6bce03e93aafc65ab2690d307 WatchSource:0}: Error finding container 1b0921cee64c7b8069550c57ef472d7b66fd71d6bce03e93aafc65ab2690d307: Status 404 returned error can't find the container with id 1b0921cee64c7b8069550c57ef472d7b66fd71d6bce03e93aafc65ab2690d307 Jul 02 19:40:00.832498 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.832459 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" event={"ID":"af9ca157-75e5-4fab-886e-5e0ce7eafb7d","Type":"ContainerStarted","Data":"da37f2f498a7e9d60c352e07ddbd7003ba2328bb0450477961e1ed7b211ea256"} Jul 02 19:40:00.832498 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.832504 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" event={"ID":"af9ca157-75e5-4fab-886e-5e0ce7eafb7d","Type":"ContainerStarted","Data":"1b0921cee64c7b8069550c57ef472d7b66fd71d6bce03e93aafc65ab2690d307"} Jul 02 19:40:00.852202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:00.852142 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" podStartSLOduration=1.852122963 podStartE2EDuration="1.852122963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:40:00.850132944 +0000 UTC m=+1939.965484193" watchObservedRunningTime="2026-07-02 19:40:00.852122963 +0000 UTC m=+1939.967474186" Jul 02 19:40:03.289691 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:03.289654 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:40:10.320540 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:10.320499 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:10.320941 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:10.320586 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:10.333027 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:10.333001 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:10.890371 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:10.890327 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:40:14.203949 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:14.203910 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:40:28.203520 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.203478 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj"] Jul 02 19:40:28.206797 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.206779 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.209426 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.209394 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cc922a\"/\"openshift-service-ca.crt\"" Jul 02 19:40:28.210431 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.210411 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cc922a\"/\"kube-root-ca.crt\"" Jul 02 19:40:28.210539 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.210518 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cc922a\"/\"default-dockercfg-crh47\"" Jul 02 19:40:28.210606 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.210551 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cc922a\"/\"test-model-cc922a-kserve-self-signed-certs\"" Jul 02 19:40:28.217572 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.217543 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj"] Jul 02 19:40:28.359189 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.359149 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-dshm\") pod \"test-model-cc922a-kserve-5d977c48d5-tg2vj\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.359397 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.359198 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-model-cache\") pod \"test-model-cc922a-kserve-5d977c48d5-tg2vj\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.359397 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.359300 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-home\") pod \"test-model-cc922a-kserve-5d977c48d5-tg2vj\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.359397 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.359330 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-tls-certs\") pod \"test-model-cc922a-kserve-5d977c48d5-tg2vj\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.359397 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.359376 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-tmp-dir\") pod \"test-model-cc922a-kserve-5d977c48d5-tg2vj\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.359564 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.359403 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlrg9\" (UniqueName: \"kubernetes.io/projected/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-kube-api-access-mlrg9\") pod \"test-model-cc922a-kserve-5d977c48d5-tg2vj\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.460831 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.460733 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-tls-certs\") pod \"test-model-cc922a-kserve-5d977c48d5-tg2vj\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.460831 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.460775 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-tmp-dir\") pod \"test-model-cc922a-kserve-5d977c48d5-tg2vj\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.460831 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.460794 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlrg9\" (UniqueName: \"kubernetes.io/projected/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-kube-api-access-mlrg9\") pod \"test-model-cc922a-kserve-5d977c48d5-tg2vj\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.460831 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.460834 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-dshm\") pod \"test-model-cc922a-kserve-5d977c48d5-tg2vj\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.461170 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.460869 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-model-cache\") pod \"test-model-cc922a-kserve-5d977c48d5-tg2vj\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.461170 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.460938 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-home\") pod \"test-model-cc922a-kserve-5d977c48d5-tg2vj\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.461170 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.461151 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-tmp-dir\") pod \"test-model-cc922a-kserve-5d977c48d5-tg2vj\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.461324 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.461265 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-home\") pod \"test-model-cc922a-kserve-5d977c48d5-tg2vj\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.461324 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.461313 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-model-cache\") pod \"test-model-cc922a-kserve-5d977c48d5-tg2vj\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.463220 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.463193 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-dshm\") pod \"test-model-cc922a-kserve-5d977c48d5-tg2vj\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.463368 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.463324 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-tls-certs\") pod \"test-model-cc922a-kserve-5d977c48d5-tg2vj\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.470290 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.470265 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlrg9\" (UniqueName: \"kubernetes.io/projected/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-kube-api-access-mlrg9\") pod \"test-model-cc922a-kserve-5d977c48d5-tg2vj\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.517230 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.517193 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:28.653717 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.653686 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj"] Jul 02 19:40:28.656735 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:40:28.656707 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a70e5a5_8b46_4d5e_8fc9_942eb495472c.slice/crio-b3ba8a576dc913bc1d804ae9f6c8330fa13921c74645f20efef5ea2d8dbb6a6b WatchSource:0}: Error finding container b3ba8a576dc913bc1d804ae9f6c8330fa13921c74645f20efef5ea2d8dbb6a6b: Status 404 returned error can't find the container with id b3ba8a576dc913bc1d804ae9f6c8330fa13921c74645f20efef5ea2d8dbb6a6b Jul 02 19:40:28.967514 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.967474 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" event={"ID":"4a70e5a5-8b46-4d5e-8fc9-942eb495472c","Type":"ContainerStarted","Data":"5b5331b7297524bb8ed9fe746c341ce70ba0da82e5d1b574962ce02874d63047"} Jul 02 19:40:28.967688 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.967523 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" event={"ID":"4a70e5a5-8b46-4d5e-8fc9-942eb495472c","Type":"ContainerStarted","Data":"b3ba8a576dc913bc1d804ae9f6c8330fa13921c74645f20efef5ea2d8dbb6a6b"} Jul 02 19:40:28.986401 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:28.986264 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" podStartSLOduration=0.986245068 podStartE2EDuration="986.245068ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:40:28.986063233 +0000 UTC m=+1968.101414457" watchObservedRunningTime="2026-07-02 19:40:28.986245068 +0000 UTC m=+1968.101596306" Jul 02 19:40:32.396562 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:32.396527 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:40:38.518331 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:38.518221 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:38.518331 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:38.518276 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:38.531011 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:38.530982 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:39.022310 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:39.022277 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:40:49.991074 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:49.991037 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh"] Jul 02 19:40:51.069472 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:51.069438 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj"] Jul 02 19:40:55.714371 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:40:55.714319 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:41:05.011200 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.011159 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" podUID="af9ca157-75e5-4fab-886e-5e0ce7eafb7d" containerName="main" containerID="cri-o://da37f2f498a7e9d60c352e07ddbd7003ba2328bb0450477961e1ed7b211ea256" gracePeriod=45 Jul 02 19:41:05.130491 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.130461 2565 generic.go:356] "Generic (PLEG): container finished" podID="af9ca157-75e5-4fab-886e-5e0ce7eafb7d" containerID="da37f2f498a7e9d60c352e07ddbd7003ba2328bb0450477961e1ed7b211ea256" exitCode=0 Jul 02 19:41:05.130637 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.130527 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" event={"ID":"af9ca157-75e5-4fab-886e-5e0ce7eafb7d","Type":"ContainerDied","Data":"da37f2f498a7e9d60c352e07ddbd7003ba2328bb0450477961e1ed7b211ea256"} Jul 02 19:41:05.267195 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.267130 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:41:05.368818 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.368775 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-model-cache\") pod \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " Jul 02 19:41:05.369015 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.368854 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-tmp-dir\") pod \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " Jul 02 19:41:05.369015 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.368875 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-home\") pod \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " Jul 02 19:41:05.369015 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.368891 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-tls-certs\") pod \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " Jul 02 19:41:05.369015 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.368920 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rn52\" (UniqueName: \"kubernetes.io/projected/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-kube-api-access-2rn52\") pod \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " Jul 02 19:41:05.369015 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.368953 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-dshm\") pod \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\" (UID: \"af9ca157-75e5-4fab-886e-5e0ce7eafb7d\") " Jul 02 19:41:05.369271 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.369119 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-model-cache" (OuterVolumeSpecName: "model-cache") pod "af9ca157-75e5-4fab-886e-5e0ce7eafb7d" (UID: "af9ca157-75e5-4fab-886e-5e0ce7eafb7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:41:05.369271 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.369193 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "af9ca157-75e5-4fab-886e-5e0ce7eafb7d" (UID: "af9ca157-75e5-4fab-886e-5e0ce7eafb7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:41:05.369271 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.369217 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-home" (OuterVolumeSpecName: "home") pod "af9ca157-75e5-4fab-886e-5e0ce7eafb7d" (UID: "af9ca157-75e5-4fab-886e-5e0ce7eafb7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:41:05.369271 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.369256 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-model-cache\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:41:05.371309 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.371275 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-kube-api-access-2rn52" (OuterVolumeSpecName: "kube-api-access-2rn52") pod "af9ca157-75e5-4fab-886e-5e0ce7eafb7d" (UID: "af9ca157-75e5-4fab-886e-5e0ce7eafb7d"). InnerVolumeSpecName "kube-api-access-2rn52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:41:05.371436 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.371328 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "af9ca157-75e5-4fab-886e-5e0ce7eafb7d" (UID: "af9ca157-75e5-4fab-886e-5e0ce7eafb7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:41:05.371436 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.371362 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-dshm" (OuterVolumeSpecName: "dshm") pod "af9ca157-75e5-4fab-886e-5e0ce7eafb7d" (UID: "af9ca157-75e5-4fab-886e-5e0ce7eafb7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:41:05.469977 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.469930 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-home\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:41:05.469977 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.469969 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-tls-certs\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:41:05.470194 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.470002 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rn52\" (UniqueName: \"kubernetes.io/projected/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-kube-api-access-2rn52\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:41:05.470194 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.470012 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-dshm\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:41:05.470194 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:05.470021 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af9ca157-75e5-4fab-886e-5e0ce7eafb7d-tmp-dir\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:41:06.089884 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.089810 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" podUID="4a70e5a5-8b46-4d5e-8fc9-942eb495472c" containerName="main" containerID="cri-o://5b5331b7297524bb8ed9fe746c341ce70ba0da82e5d1b574962ce02874d63047" gracePeriod=45 Jul 02 19:41:06.136274 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.136247 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" Jul 02 19:41:06.136482 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.136268 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh" event={"ID":"af9ca157-75e5-4fab-886e-5e0ce7eafb7d","Type":"ContainerDied","Data":"1b0921cee64c7b8069550c57ef472d7b66fd71d6bce03e93aafc65ab2690d307"} Jul 02 19:41:06.136482 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.136320 2565 scope.go:117] "RemoveContainer" containerID="da37f2f498a7e9d60c352e07ddbd7003ba2328bb0450477961e1ed7b211ea256" Jul 02 19:41:06.163039 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.163002 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh"] Jul 02 19:41:06.167282 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.167250 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-09f1f6/test-model-09f1f6-kserve-5596df4bfc-p55zh"] Jul 02 19:41:06.358189 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.358158 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:41:06.477561 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.477530 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-model-cache\") pod \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " Jul 02 19:41:06.477561 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.477574 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-tls-certs\") pod \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " Jul 02 19:41:06.477827 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.477596 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-dshm\") pod \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " Jul 02 19:41:06.477827 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.477614 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlrg9\" (UniqueName: \"kubernetes.io/projected/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-kube-api-access-mlrg9\") pod \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " Jul 02 19:41:06.477827 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.477651 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-home\") pod \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " Jul 02 19:41:06.477827 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.477699 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-tmp-dir\") pod \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\" (UID: \"4a70e5a5-8b46-4d5e-8fc9-942eb495472c\") " Jul 02 19:41:06.478039 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.477874 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a70e5a5-8b46-4d5e-8fc9-942eb495472c" (UID: "4a70e5a5-8b46-4d5e-8fc9-942eb495472c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:41:06.478039 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.477952 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-home" (OuterVolumeSpecName: "home") pod "4a70e5a5-8b46-4d5e-8fc9-942eb495472c" (UID: "4a70e5a5-8b46-4d5e-8fc9-942eb495472c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:41:06.478133 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.478034 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a70e5a5-8b46-4d5e-8fc9-942eb495472c" (UID: "4a70e5a5-8b46-4d5e-8fc9-942eb495472c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:41:06.479875 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.479844 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a70e5a5-8b46-4d5e-8fc9-942eb495472c" (UID: "4a70e5a5-8b46-4d5e-8fc9-942eb495472c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:41:06.479979 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.479884 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-dshm" (OuterVolumeSpecName: "dshm") pod "4a70e5a5-8b46-4d5e-8fc9-942eb495472c" (UID: "4a70e5a5-8b46-4d5e-8fc9-942eb495472c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:41:06.479979 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.479895 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-kube-api-access-mlrg9" (OuterVolumeSpecName: "kube-api-access-mlrg9") pod "4a70e5a5-8b46-4d5e-8fc9-942eb495472c" (UID: "4a70e5a5-8b46-4d5e-8fc9-942eb495472c"). InnerVolumeSpecName "kube-api-access-mlrg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:41:06.578441 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.578403 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-tmp-dir\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:41:06.578441 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.578434 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-model-cache\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:41:06.578441 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.578443 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-tls-certs\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:41:06.578441 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.578452 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-dshm\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:41:06.578694 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.578460 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlrg9\" (UniqueName: \"kubernetes.io/projected/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-kube-api-access-mlrg9\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:41:06.578694 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:06.578469 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a70e5a5-8b46-4d5e-8fc9-942eb495472c-home\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:41:07.142892 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:07.142862 2565 generic.go:356] "Generic (PLEG): container finished" podID="4a70e5a5-8b46-4d5e-8fc9-942eb495472c" containerID="5b5331b7297524bb8ed9fe746c341ce70ba0da82e5d1b574962ce02874d63047" exitCode=0 Jul 02 19:41:07.143306 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:07.142902 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" Jul 02 19:41:07.143306 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:07.142912 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" event={"ID":"4a70e5a5-8b46-4d5e-8fc9-942eb495472c","Type":"ContainerDied","Data":"5b5331b7297524bb8ed9fe746c341ce70ba0da82e5d1b574962ce02874d63047"} Jul 02 19:41:07.143306 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:07.142942 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj" event={"ID":"4a70e5a5-8b46-4d5e-8fc9-942eb495472c","Type":"ContainerDied","Data":"b3ba8a576dc913bc1d804ae9f6c8330fa13921c74645f20efef5ea2d8dbb6a6b"} Jul 02 19:41:07.143306 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:07.142958 2565 scope.go:117] "RemoveContainer" containerID="5b5331b7297524bb8ed9fe746c341ce70ba0da82e5d1b574962ce02874d63047" Jul 02 19:41:07.152608 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:07.152586 2565 scope.go:117] "RemoveContainer" containerID="5b5331b7297524bb8ed9fe746c341ce70ba0da82e5d1b574962ce02874d63047" Jul 02 19:41:07.152891 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:41:07.152875 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b5331b7297524bb8ed9fe746c341ce70ba0da82e5d1b574962ce02874d63047\": container with ID starting with 5b5331b7297524bb8ed9fe746c341ce70ba0da82e5d1b574962ce02874d63047 not found: ID does not exist" containerID="5b5331b7297524bb8ed9fe746c341ce70ba0da82e5d1b574962ce02874d63047" Jul 02 19:41:07.152941 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:07.152900 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b5331b7297524bb8ed9fe746c341ce70ba0da82e5d1b574962ce02874d63047"} err="failed to get container status \"5b5331b7297524bb8ed9fe746c341ce70ba0da82e5d1b574962ce02874d63047\": rpc error: code = NotFound desc = could not find container \"5b5331b7297524bb8ed9fe746c341ce70ba0da82e5d1b574962ce02874d63047\": container with ID starting with 5b5331b7297524bb8ed9fe746c341ce70ba0da82e5d1b574962ce02874d63047 not found: ID does not exist" Jul 02 19:41:07.165410 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:07.165376 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj"] Jul 02 19:41:07.169937 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:07.169911 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cc922a/test-model-cc922a-kserve-5d977c48d5-tg2vj"] Jul 02 19:41:07.399888 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:07.399863 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a70e5a5-8b46-4d5e-8fc9-942eb495472c" path="/var/lib/kubelet/pods/4a70e5a5-8b46-4d5e-8fc9-942eb495472c/volumes" Jul 02 19:41:07.400250 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:07.400237 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af9ca157-75e5-4fab-886e-5e0ce7eafb7d" path="/var/lib/kubelet/pods/af9ca157-75e5-4fab-886e-5e0ce7eafb7d/volumes" Jul 02 19:41:37.909164 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:37.909119 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:41:50.894842 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:41:50.894802 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:42:07.213738 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:07.213646 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:42:18.401767 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:18.401726 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:42:29.510416 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:29.510376 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:42:41.456807 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:41.456776 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:42:41.459066 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:41.459040 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:42:45.094248 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:45.094214 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:42:50.091938 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.091898 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9"] Jul 02 19:42:50.092426 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.092183 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af9ca157-75e5-4fab-886e-5e0ce7eafb7d" containerName="main" Jul 02 19:42:50.092426 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.092194 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="af9ca157-75e5-4fab-886e-5e0ce7eafb7d" containerName="main" Jul 02 19:42:50.092426 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.092215 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a70e5a5-8b46-4d5e-8fc9-942eb495472c" containerName="main" Jul 02 19:42:50.092426 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.092220 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a70e5a5-8b46-4d5e-8fc9-942eb495472c" containerName="main" Jul 02 19:42:50.092426 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.092271 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a70e5a5-8b46-4d5e-8fc9-942eb495472c" containerName="main" Jul 02 19:42:50.092426 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.092279 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="af9ca157-75e5-4fab-886e-5e0ce7eafb7d" containerName="main" Jul 02 19:42:50.095079 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.095059 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.099366 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.099317 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-67cf22\"/\"kube-root-ca.crt\"" Jul 02 19:42:50.099487 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.099317 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-67cf22\"/\"auth-test-model-67cf22-kserve-self-signed-certs\"" Jul 02 19:42:50.099487 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.099395 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-67cf22\"/\"openshift-service-ca.crt\"" Jul 02 19:42:50.099487 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.099323 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-67cf22\"/\"default-dockercfg-frrbz\"" Jul 02 19:42:50.107672 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.107646 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9"] Jul 02 19:42:50.135087 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.135050 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhjvm\" (UniqueName: \"kubernetes.io/projected/8d4a7075-b92b-4942-83c0-e664df1278c0-kube-api-access-mhjvm\") pod \"auth-test-model-67cf22-kserve-596d796fc9-5v8v9\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.135282 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.135096 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-model-cache\") pod \"auth-test-model-67cf22-kserve-596d796fc9-5v8v9\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.135282 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.135158 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-dshm\") pod \"auth-test-model-67cf22-kserve-596d796fc9-5v8v9\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.135282 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.135200 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-tmp-dir\") pod \"auth-test-model-67cf22-kserve-596d796fc9-5v8v9\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.135282 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.135257 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d4a7075-b92b-4942-83c0-e664df1278c0-tls-certs\") pod \"auth-test-model-67cf22-kserve-596d796fc9-5v8v9\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.135469 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.135287 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-home\") pod \"auth-test-model-67cf22-kserve-596d796fc9-5v8v9\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.235924 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.235878 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhjvm\" (UniqueName: \"kubernetes.io/projected/8d4a7075-b92b-4942-83c0-e664df1278c0-kube-api-access-mhjvm\") pod \"auth-test-model-67cf22-kserve-596d796fc9-5v8v9\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.236123 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.235946 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-model-cache\") pod \"auth-test-model-67cf22-kserve-596d796fc9-5v8v9\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.236123 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.235982 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-dshm\") pod \"auth-test-model-67cf22-kserve-596d796fc9-5v8v9\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.236123 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.236007 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-tmp-dir\") pod \"auth-test-model-67cf22-kserve-596d796fc9-5v8v9\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.236123 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.236054 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d4a7075-b92b-4942-83c0-e664df1278c0-tls-certs\") pod \"auth-test-model-67cf22-kserve-596d796fc9-5v8v9\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.236123 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.236087 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-home\") pod \"auth-test-model-67cf22-kserve-596d796fc9-5v8v9\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.236526 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.236500 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-tmp-dir\") pod \"auth-test-model-67cf22-kserve-596d796fc9-5v8v9\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.236602 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.236523 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-model-cache\") pod \"auth-test-model-67cf22-kserve-596d796fc9-5v8v9\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.236602 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.236561 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-home\") pod \"auth-test-model-67cf22-kserve-596d796fc9-5v8v9\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.238273 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.238252 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-dshm\") pod \"auth-test-model-67cf22-kserve-596d796fc9-5v8v9\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.238582 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.238565 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d4a7075-b92b-4942-83c0-e664df1278c0-tls-certs\") pod \"auth-test-model-67cf22-kserve-596d796fc9-5v8v9\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.244600 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.244572 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhjvm\" (UniqueName: \"kubernetes.io/projected/8d4a7075-b92b-4942-83c0-e664df1278c0-kube-api-access-mhjvm\") pod \"auth-test-model-67cf22-kserve-596d796fc9-5v8v9\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.403612 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.403576 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:42:50.545185 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.545025 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9"] Jul 02 19:42:50.547875 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:42:50.547839 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d4a7075_b92b_4942_83c0_e664df1278c0.slice/crio-59419763c51e57ca32df0525e0049620e3c18a1f98031947ace2706eef882696 WatchSource:0}: Error finding container 59419763c51e57ca32df0525e0049620e3c18a1f98031947ace2706eef882696: Status 404 returned error can't find the container with id 59419763c51e57ca32df0525e0049620e3c18a1f98031947ace2706eef882696 Jul 02 19:42:50.584503 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:50.584471 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" event={"ID":"8d4a7075-b92b-4942-83c0-e664df1278c0","Type":"ContainerStarted","Data":"59419763c51e57ca32df0525e0049620e3c18a1f98031947ace2706eef882696"} Jul 02 19:42:51.595188 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:51.595148 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" event={"ID":"8d4a7075-b92b-4942-83c0-e664df1278c0","Type":"ContainerStarted","Data":"d3442036f5e345eb238d7b73db7b411cf3d31f7ca240cc5b1a6ff912e596a9fb"} Jul 02 19:42:51.615963 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:51.615909 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" podStartSLOduration=1.615892102 podStartE2EDuration="1.615892102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:42:51.614602677 +0000 UTC m=+2110.729953908" watchObservedRunningTime="2026-07-02 19:42:51.615892102 +0000 UTC m=+2110.731243338" Jul 02 19:42:54.096637 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:42:54.096597 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:43:00.404546 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:00.404500 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:43:00.404546 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:00.404550 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:43:00.417698 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:00.417657 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:43:00.644184 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:00.644153 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:43:36.001548 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.001466 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj"] Jul 02 19:43:36.004732 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.004715 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.008272 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.008246 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0c31dc\"/\"openshift-service-ca.crt\"" Jul 02 19:43:36.008427 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.008278 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0c31dc\"/\"kube-root-ca.crt\"" Jul 02 19:43:36.008427 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.008250 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0c31dc\"/\"default-dockercfg-g6cjj\"" Jul 02 19:43:36.008427 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.008250 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0c31dc\"/\"auth-test-model-0c31dc-kserve-self-signed-certs\"" Jul 02 19:43:36.014160 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.014135 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj"] Jul 02 19:43:36.138268 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.138221 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-model-cache\") pod \"auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.138268 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.138276 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-dshm\") pod \"auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.138541 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.138294 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qptzz\" (UniqueName: \"kubernetes.io/projected/c5f65e30-e8d4-4991-ae51-44494f7376fb-kube-api-access-qptzz\") pod \"auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.138541 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.138417 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-home\") pod \"auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.138541 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.138465 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-tmp-dir\") pod \"auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.138541 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.138483 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f65e30-e8d4-4991-ae51-44494f7376fb-tls-certs\") pod \"auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.239904 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.239867 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-tmp-dir\") pod \"auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.239904 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.239906 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f65e30-e8d4-4991-ae51-44494f7376fb-tls-certs\") pod \"auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.240137 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.239952 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-model-cache\") pod \"auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.240137 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.239976 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-dshm\") pod \"auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.240137 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.239993 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qptzz\" (UniqueName: \"kubernetes.io/projected/c5f65e30-e8d4-4991-ae51-44494f7376fb-kube-api-access-qptzz\") pod \"auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.240268 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.240130 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-home\") pod \"auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.240325 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.240308 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-tmp-dir\") pod \"auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.240532 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.240486 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-model-cache\") pod \"auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.240532 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.240501 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-home\") pod \"auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.242294 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.242272 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-dshm\") pod \"auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.242579 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.242559 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f65e30-e8d4-4991-ae51-44494f7376fb-tls-certs\") pod \"auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.248071 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.248047 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qptzz\" (UniqueName: \"kubernetes.io/projected/c5f65e30-e8d4-4991-ae51-44494f7376fb-kube-api-access-qptzz\") pod \"auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.314238 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.314144 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:36.452676 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.452648 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj"] Jul 02 19:43:36.455708 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:43:36.455680 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5f65e30_e8d4_4991_ae51_44494f7376fb.slice/crio-5fc1c2686a4e281884c63dd6f84127edb4bd0c5d2260d49795bf754ddc69fcf2 WatchSource:0}: Error finding container 5fc1c2686a4e281884c63dd6f84127edb4bd0c5d2260d49795bf754ddc69fcf2: Status 404 returned error can't find the container with id 5fc1c2686a4e281884c63dd6f84127edb4bd0c5d2260d49795bf754ddc69fcf2 Jul 02 19:43:36.792875 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.792831 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" event={"ID":"c5f65e30-e8d4-4991-ae51-44494f7376fb","Type":"ContainerStarted","Data":"9e6eb682212328c13035725914e444600c390db1287247a0160c04fcb64b3c45"} Jul 02 19:43:36.793052 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.792885 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" event={"ID":"c5f65e30-e8d4-4991-ae51-44494f7376fb","Type":"ContainerStarted","Data":"5fc1c2686a4e281884c63dd6f84127edb4bd0c5d2260d49795bf754ddc69fcf2"} Jul 02 19:43:36.811321 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:36.811267 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" podStartSLOduration=1.811249318 podStartE2EDuration="1.811249318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:43:36.810948393 +0000 UTC m=+2155.926299618" watchObservedRunningTime="2026-07-02 19:43:36.811249318 +0000 UTC m=+2155.926600540" Jul 02 19:43:39.609174 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:39.609138 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:43:46.314416 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:46.314382 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:46.316737 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:46.314431 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:46.326754 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:46.326726 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:43:46.849500 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:43:46.849462 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:44:00.301440 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:44:00.301398 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:44:18.199906 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:44:18.199866 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:44:29.687597 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:44:29.687558 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:44:46.094032 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:44:46.093996 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:45:00.130263 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:00.130218 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h"] Jul 02 19:45:00.133885 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:00.133864 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4"] Jul 02 19:45:00.134005 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:00.133987 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h" Jul 02 19:45:00.136601 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:00.136581 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-rtxc5\"" Jul 02 19:45:00.137196 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:00.137178 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4" Jul 02 19:45:00.140827 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:00.140799 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4"] Jul 02 19:45:00.144415 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:00.144390 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h"] Jul 02 19:45:00.286772 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:00.286718 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kvxd\" (UniqueName: \"kubernetes.io/projected/e8af756a-e839-4346-95f6-145990b3a6f8-kube-api-access-8kvxd\") pod \"maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4\" (UID: \"e8af756a-e839-4346-95f6-145990b3a6f8\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4" Jul 02 19:45:00.286958 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:00.286830 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75rtz\" (UniqueName: \"kubernetes.io/projected/f999c236-0aa1-44f7-b04f-a0bb33ab1a15-kube-api-access-75rtz\") pod \"maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h\" (UID: \"f999c236-0aa1-44f7-b04f-a0bb33ab1a15\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h" Jul 02 19:45:00.388155 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:00.388065 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kvxd\" (UniqueName: \"kubernetes.io/projected/e8af756a-e839-4346-95f6-145990b3a6f8-kube-api-access-8kvxd\") pod \"maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4\" (UID: \"e8af756a-e839-4346-95f6-145990b3a6f8\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4" Jul 02 19:45:00.388155 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:00.388120 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75rtz\" (UniqueName: \"kubernetes.io/projected/f999c236-0aa1-44f7-b04f-a0bb33ab1a15-kube-api-access-75rtz\") pod \"maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h\" (UID: \"f999c236-0aa1-44f7-b04f-a0bb33ab1a15\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h" Jul 02 19:45:00.397180 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:00.397149 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kvxd\" (UniqueName: \"kubernetes.io/projected/e8af756a-e839-4346-95f6-145990b3a6f8-kube-api-access-8kvxd\") pod \"maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4\" (UID: \"e8af756a-e839-4346-95f6-145990b3a6f8\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4" Jul 02 19:45:00.397304 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:00.397251 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75rtz\" (UniqueName: \"kubernetes.io/projected/f999c236-0aa1-44f7-b04f-a0bb33ab1a15-kube-api-access-75rtz\") pod \"maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h\" (UID: \"f999c236-0aa1-44f7-b04f-a0bb33ab1a15\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h" Jul 02 19:45:00.448003 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:00.447956 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h" Jul 02 19:45:00.455016 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:00.454994 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4" Jul 02 19:45:00.796146 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:00.796120 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h"] Jul 02 19:45:00.798411 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:45:00.798383 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf999c236_0aa1_44f7_b04f_a0bb33ab1a15.slice/crio-23e510a90d09cabc01052ac266efb4db745118b76b15e8abd85939cc34e38799 WatchSource:0}: Error finding container 23e510a90d09cabc01052ac266efb4db745118b76b15e8abd85939cc34e38799: Status 404 returned error can't find the container with id 23e510a90d09cabc01052ac266efb4db745118b76b15e8abd85939cc34e38799 Jul 02 19:45:00.800322 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:00.800304 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:45:00.826386 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:00.826246 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4"] Jul 02 19:45:00.828528 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:45:00.828497 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8af756a_e839_4346_95f6_145990b3a6f8.slice/crio-9d6cda8e164408b3b39ccfc29ec6f08425a526a0a08293161daff005ccbdbef3 WatchSource:0}: Error finding container 9d6cda8e164408b3b39ccfc29ec6f08425a526a0a08293161daff005ccbdbef3: Status 404 returned error can't find the container with id 9d6cda8e164408b3b39ccfc29ec6f08425a526a0a08293161daff005ccbdbef3 Jul 02 19:45:01.162643 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:01.162604 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4" event={"ID":"e8af756a-e839-4346-95f6-145990b3a6f8","Type":"ContainerStarted","Data":"9d6cda8e164408b3b39ccfc29ec6f08425a526a0a08293161daff005ccbdbef3"} Jul 02 19:45:01.163621 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:01.163597 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h" event={"ID":"f999c236-0aa1-44f7-b04f-a0bb33ab1a15","Type":"ContainerStarted","Data":"23e510a90d09cabc01052ac266efb4db745118b76b15e8abd85939cc34e38799"} Jul 02 19:45:02.007547 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:02.007503 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:45:04.180851 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:04.180816 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4" event={"ID":"e8af756a-e839-4346-95f6-145990b3a6f8","Type":"ContainerStarted","Data":"42209677eb287ecb4f4ca7d3b70af78fb314f7ca123864344cec51081b354161"} Jul 02 19:45:04.182281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:04.182257 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h" event={"ID":"f999c236-0aa1-44f7-b04f-a0bb33ab1a15","Type":"ContainerStarted","Data":"651d6d20252bfce8b48f5b73fc3404ef5b23b0e616e02d2fab7ccaa9f0221d30"} Jul 02 19:45:04.198765 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:04.198709 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4" podStartSLOduration=1.728571794 podStartE2EDuration="4.198694461s" totalImagesPullingTime="2.470122667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:45:04.196493943 +0000 UTC m=+2243.311845192" watchObservedRunningTime="2026-07-02 19:45:04.198694461 +0000 UTC m=+2243.314045661" Jul 02 19:45:04.215538 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:04.215485 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h" podStartSLOduration=1.720669665 podStartE2EDuration="4.215465674s" totalImagesPullingTime="2.494796009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:45:04.213614362 +0000 UTC m=+2243.328965609" watchObservedRunningTime="2026-07-02 19:45:04.215465674 +0000 UTC m=+2243.330816897" Jul 02 19:45:05.188084 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:05.188052 2565 generic.go:356] "Generic (PLEG): container finished" podID="e8af756a-e839-4346-95f6-145990b3a6f8" containerID="42209677eb287ecb4f4ca7d3b70af78fb314f7ca123864344cec51081b354161" exitCode=0 Jul 02 19:45:05.188517 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:05.188145 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4" event={"ID":"e8af756a-e839-4346-95f6-145990b3a6f8","Type":"ContainerDied","Data":"42209677eb287ecb4f4ca7d3b70af78fb314f7ca123864344cec51081b354161"} Jul 02 19:45:05.189620 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:05.189601 2565 generic.go:356] "Generic (PLEG): container finished" podID="f999c236-0aa1-44f7-b04f-a0bb33ab1a15" containerID="651d6d20252bfce8b48f5b73fc3404ef5b23b0e616e02d2fab7ccaa9f0221d30" exitCode=0 Jul 02 19:45:05.189708 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:05.189638 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h" event={"ID":"f999c236-0aa1-44f7-b04f-a0bb33ab1a15","Type":"ContainerDied","Data":"651d6d20252bfce8b48f5b73fc3404ef5b23b0e616e02d2fab7ccaa9f0221d30"} Jul 02 19:45:06.326457 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:06.326429 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4" Jul 02 19:45:06.369124 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:06.369095 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h" Jul 02 19:45:06.438851 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:06.438810 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kvxd\" (UniqueName: \"kubernetes.io/projected/e8af756a-e839-4346-95f6-145990b3a6f8-kube-api-access-8kvxd\") pod \"e8af756a-e839-4346-95f6-145990b3a6f8\" (UID: \"e8af756a-e839-4346-95f6-145990b3a6f8\") " Jul 02 19:45:06.441050 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:06.441014 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8af756a-e839-4346-95f6-145990b3a6f8-kube-api-access-8kvxd" (OuterVolumeSpecName: "kube-api-access-8kvxd") pod "e8af756a-e839-4346-95f6-145990b3a6f8" (UID: "e8af756a-e839-4346-95f6-145990b3a6f8"). InnerVolumeSpecName "kube-api-access-8kvxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:45:06.539800 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:06.539757 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75rtz\" (UniqueName: \"kubernetes.io/projected/f999c236-0aa1-44f7-b04f-a0bb33ab1a15-kube-api-access-75rtz\") pod \"f999c236-0aa1-44f7-b04f-a0bb33ab1a15\" (UID: \"f999c236-0aa1-44f7-b04f-a0bb33ab1a15\") " Jul 02 19:45:06.540050 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:06.540033 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kvxd\" (UniqueName: \"kubernetes.io/projected/e8af756a-e839-4346-95f6-145990b3a6f8-kube-api-access-8kvxd\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:45:06.541848 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:06.541818 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f999c236-0aa1-44f7-b04f-a0bb33ab1a15-kube-api-access-75rtz" (OuterVolumeSpecName: "kube-api-access-75rtz") pod "f999c236-0aa1-44f7-b04f-a0bb33ab1a15" (UID: "f999c236-0aa1-44f7-b04f-a0bb33ab1a15"). InnerVolumeSpecName "kube-api-access-75rtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:45:06.640477 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:06.640441 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75rtz\" (UniqueName: \"kubernetes.io/projected/f999c236-0aa1-44f7-b04f-a0bb33ab1a15-kube-api-access-75rtz\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:45:07.200087 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:07.200057 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h" Jul 02 19:45:07.200280 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:07.200054 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h" event={"ID":"f999c236-0aa1-44f7-b04f-a0bb33ab1a15","Type":"ContainerDied","Data":"23e510a90d09cabc01052ac266efb4db745118b76b15e8abd85939cc34e38799"} Jul 02 19:45:07.200280 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:07.200201 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23e510a90d09cabc01052ac266efb4db745118b76b15e8abd85939cc34e38799" Jul 02 19:45:07.201392 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:07.201368 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4" event={"ID":"e8af756a-e839-4346-95f6-145990b3a6f8","Type":"ContainerDied","Data":"9d6cda8e164408b3b39ccfc29ec6f08425a526a0a08293161daff005ccbdbef3"} Jul 02 19:45:07.201491 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:07.201412 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d6cda8e164408b3b39ccfc29ec6f08425a526a0a08293161daff005ccbdbef3" Jul 02 19:45:07.201491 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:07.201382 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4" Jul 02 19:45:18.415313 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:18.415263 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:45:35.289628 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:35.289587 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:45:39.173293 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:39.173248 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9"] Jul 02 19:45:40.180448 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:40.180410 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj"] Jul 02 19:45:51.840733 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:51.840695 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:45:51.992717 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:51.992684 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x"] Jul 02 19:45:51.992991 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:51.992979 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f999c236-0aa1-44f7-b04f-a0bb33ab1a15" containerName="cleanup" Jul 02 19:45:51.993039 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:51.992992 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f999c236-0aa1-44f7-b04f-a0bb33ab1a15" containerName="cleanup" Jul 02 19:45:51.993039 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:51.993011 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8af756a-e839-4346-95f6-145990b3a6f8" containerName="cleanup" Jul 02 19:45:51.993039 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:51.993016 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8af756a-e839-4346-95f6-145990b3a6f8" containerName="cleanup" Jul 02 19:45:51.993133 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:51.993075 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8af756a-e839-4346-95f6-145990b3a6f8" containerName="cleanup" Jul 02 19:45:51.993133 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:51.993084 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f999c236-0aa1-44f7-b04f-a0bb33ab1a15" containerName="cleanup" Jul 02 19:45:51.995871 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:51.995852 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:51.999451 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:51.999416 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-67cf22\"/\"sub-test-model-67cf22-kserve-self-signed-certs\"" Jul 02 19:45:52.006380 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.006334 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x"] Jul 02 19:45:52.135106 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.135005 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-dshm\") pod \"sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:52.135106 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.135068 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-model-cache\") pod \"sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:52.135403 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.135177 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-tmp-dir\") pod \"sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:52.135403 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.135212 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d6337a-b571-4960-82ee-935cef24d4dc-tls-certs\") pod \"sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:52.135403 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.135272 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8qzh\" (UniqueName: \"kubernetes.io/projected/c7d6337a-b571-4960-82ee-935cef24d4dc-kube-api-access-x8qzh\") pod \"sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:52.135403 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.135338 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-home\") pod \"sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:52.236493 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.236449 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-home\") pod \"sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:52.236663 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.236519 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-dshm\") pod \"sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:52.236663 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.236567 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-model-cache\") pod \"sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:52.236663 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.236613 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-tmp-dir\") pod \"sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:52.236663 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.236636 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d6337a-b571-4960-82ee-935cef24d4dc-tls-certs\") pod \"sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:52.236884 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.236671 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8qzh\" (UniqueName: \"kubernetes.io/projected/c7d6337a-b571-4960-82ee-935cef24d4dc-kube-api-access-x8qzh\") pod \"sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:52.236942 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.236892 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-home\") pod \"sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:52.237078 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.237056 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-model-cache\") pod \"sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:52.237129 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.237074 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-tmp-dir\") pod \"sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:52.238799 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.238780 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-dshm\") pod \"sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:52.239097 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.239078 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d6337a-b571-4960-82ee-935cef24d4dc-tls-certs\") pod \"sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:52.250647 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.250622 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8qzh\" (UniqueName: \"kubernetes.io/projected/c7d6337a-b571-4960-82ee-935cef24d4dc-kube-api-access-x8qzh\") pod \"sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:52.305751 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.305707 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:45:52.441013 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:52.440991 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x"] Jul 02 19:45:52.443231 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:45:52.443202 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7d6337a_b571_4960_82ee_935cef24d4dc.slice/crio-e6072bba9d02b740f2cc6e75db14387f84ef4a9017278321a7770e659b698b9a WatchSource:0}: Error finding container e6072bba9d02b740f2cc6e75db14387f84ef4a9017278321a7770e659b698b9a: Status 404 returned error can't find the container with id e6072bba9d02b740f2cc6e75db14387f84ef4a9017278321a7770e659b698b9a Jul 02 19:45:53.409252 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:53.409220 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" event={"ID":"c7d6337a-b571-4960-82ee-935cef24d4dc","Type":"ContainerStarted","Data":"f38aa4722649b17cabc2e19ae324293496fd0f7666ce3b157d9ea763adb8cdeb"} Jul 02 19:45:53.409252 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:53.409258 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" event={"ID":"c7d6337a-b571-4960-82ee-935cef24d4dc","Type":"ContainerStarted","Data":"e6072bba9d02b740f2cc6e75db14387f84ef4a9017278321a7770e659b698b9a"} Jul 02 19:45:53.427914 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:53.427859 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" podStartSLOduration=2.427842227 podStartE2EDuration="2.427842227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:45:53.425930253 +0000 UTC m=+2292.541281476" watchObservedRunningTime="2026-07-02 19:45:53.427842227 +0000 UTC m=+2292.543193450" Jul 02 19:45:54.193794 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.193744 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" podUID="8d4a7075-b92b-4942-83c0-e664df1278c0" containerName="main" containerID="cri-o://d3442036f5e345eb238d7b73db7b411cf3d31f7ca240cc5b1a6ff912e596a9fb" gracePeriod=45 Jul 02 19:45:54.415709 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.415679 2565 generic.go:356] "Generic (PLEG): container finished" podID="8d4a7075-b92b-4942-83c0-e664df1278c0" containerID="d3442036f5e345eb238d7b73db7b411cf3d31f7ca240cc5b1a6ff912e596a9fb" exitCode=0 Jul 02 19:45:54.416170 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.415755 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" event={"ID":"8d4a7075-b92b-4942-83c0-e664df1278c0","Type":"ContainerDied","Data":"d3442036f5e345eb238d7b73db7b411cf3d31f7ca240cc5b1a6ff912e596a9fb"} Jul 02 19:45:54.449096 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.449041 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:45:54.559124 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.559084 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d4a7075-b92b-4942-83c0-e664df1278c0-tls-certs\") pod \"8d4a7075-b92b-4942-83c0-e664df1278c0\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " Jul 02 19:45:54.559327 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.559155 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-tmp-dir\") pod \"8d4a7075-b92b-4942-83c0-e664df1278c0\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " Jul 02 19:45:54.559327 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.559210 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-model-cache\") pod \"8d4a7075-b92b-4942-83c0-e664df1278c0\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " Jul 02 19:45:54.559327 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.559245 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhjvm\" (UniqueName: \"kubernetes.io/projected/8d4a7075-b92b-4942-83c0-e664df1278c0-kube-api-access-mhjvm\") pod \"8d4a7075-b92b-4942-83c0-e664df1278c0\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " Jul 02 19:45:54.559327 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.559285 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-home\") pod \"8d4a7075-b92b-4942-83c0-e664df1278c0\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " Jul 02 19:45:54.559327 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.559312 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-dshm\") pod \"8d4a7075-b92b-4942-83c0-e664df1278c0\" (UID: \"8d4a7075-b92b-4942-83c0-e664df1278c0\") " Jul 02 19:45:54.559620 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.559524 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d4a7075-b92b-4942-83c0-e664df1278c0" (UID: "8d4a7075-b92b-4942-83c0-e664df1278c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:45:54.559620 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.559536 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d4a7075-b92b-4942-83c0-e664df1278c0" (UID: "8d4a7075-b92b-4942-83c0-e664df1278c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:45:54.559620 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.559594 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-home" (OuterVolumeSpecName: "home") pod "8d4a7075-b92b-4942-83c0-e664df1278c0" (UID: "8d4a7075-b92b-4942-83c0-e664df1278c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:45:54.560116 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.560041 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-tmp-dir\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:45:54.560116 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.560068 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-model-cache\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:45:54.560116 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.560082 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-home\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:45:54.561587 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.561565 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d4a7075-b92b-4942-83c0-e664df1278c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d4a7075-b92b-4942-83c0-e664df1278c0" (UID: "8d4a7075-b92b-4942-83c0-e664df1278c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:45:54.561587 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.561573 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4a7075-b92b-4942-83c0-e664df1278c0-kube-api-access-mhjvm" (OuterVolumeSpecName: "kube-api-access-mhjvm") pod "8d4a7075-b92b-4942-83c0-e664df1278c0" (UID: "8d4a7075-b92b-4942-83c0-e664df1278c0"). InnerVolumeSpecName "kube-api-access-mhjvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:45:54.561693 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.561628 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-dshm" (OuterVolumeSpecName: "dshm") pod "8d4a7075-b92b-4942-83c0-e664df1278c0" (UID: "8d4a7075-b92b-4942-83c0-e664df1278c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:45:54.660762 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.660715 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d4a7075-b92b-4942-83c0-e664df1278c0-tls-certs\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:45:54.660762 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.660754 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhjvm\" (UniqueName: \"kubernetes.io/projected/8d4a7075-b92b-4942-83c0-e664df1278c0-kube-api-access-mhjvm\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:45:54.660762 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:54.660766 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d4a7075-b92b-4942-83c0-e664df1278c0-dshm\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:45:55.201532 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.201488 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" podUID="c5f65e30-e8d4-4991-ae51-44494f7376fb" containerName="main" containerID="cri-o://9e6eb682212328c13035725914e444600c390db1287247a0160c04fcb64b3c45" gracePeriod=45 Jul 02 19:45:55.421446 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.421414 2565 generic.go:356] "Generic (PLEG): container finished" podID="c5f65e30-e8d4-4991-ae51-44494f7376fb" containerID="9e6eb682212328c13035725914e444600c390db1287247a0160c04fcb64b3c45" exitCode=0 Jul 02 19:45:55.421881 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.421489 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" event={"ID":"c5f65e30-e8d4-4991-ae51-44494f7376fb","Type":"ContainerDied","Data":"9e6eb682212328c13035725914e444600c390db1287247a0160c04fcb64b3c45"} Jul 02 19:45:55.422761 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.422740 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" event={"ID":"8d4a7075-b92b-4942-83c0-e664df1278c0","Type":"ContainerDied","Data":"59419763c51e57ca32df0525e0049620e3c18a1f98031947ace2706eef882696"} Jul 02 19:45:55.422761 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.422753 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9" Jul 02 19:45:55.422926 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.422779 2565 scope.go:117] "RemoveContainer" containerID="d3442036f5e345eb238d7b73db7b411cf3d31f7ca240cc5b1a6ff912e596a9fb" Jul 02 19:45:55.439169 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.439138 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9"] Jul 02 19:45:55.442258 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.442233 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-67cf22/auth-test-model-67cf22-kserve-596d796fc9-5v8v9"] Jul 02 19:45:55.457816 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.457748 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:45:55.568860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.568819 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f65e30-e8d4-4991-ae51-44494f7376fb-tls-certs\") pod \"c5f65e30-e8d4-4991-ae51-44494f7376fb\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " Jul 02 19:45:55.569059 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.568891 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-tmp-dir\") pod \"c5f65e30-e8d4-4991-ae51-44494f7376fb\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " Jul 02 19:45:55.569059 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.568920 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-home\") pod \"c5f65e30-e8d4-4991-ae51-44494f7376fb\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " Jul 02 19:45:55.569059 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.568958 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qptzz\" (UniqueName: \"kubernetes.io/projected/c5f65e30-e8d4-4991-ae51-44494f7376fb-kube-api-access-qptzz\") pod \"c5f65e30-e8d4-4991-ae51-44494f7376fb\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " Jul 02 19:45:55.569059 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.568996 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-model-cache\") pod \"c5f65e30-e8d4-4991-ae51-44494f7376fb\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " Jul 02 19:45:55.569059 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.569046 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-dshm\") pod \"c5f65e30-e8d4-4991-ae51-44494f7376fb\" (UID: \"c5f65e30-e8d4-4991-ae51-44494f7376fb\") " Jul 02 19:45:55.569297 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.569208 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c5f65e30-e8d4-4991-ae51-44494f7376fb" (UID: "c5f65e30-e8d4-4991-ae51-44494f7376fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:45:55.569297 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.569218 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-home" (OuterVolumeSpecName: "home") pod "c5f65e30-e8d4-4991-ae51-44494f7376fb" (UID: "c5f65e30-e8d4-4991-ae51-44494f7376fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:45:55.569297 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.569266 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "c5f65e30-e8d4-4991-ae51-44494f7376fb" (UID: "c5f65e30-e8d4-4991-ae51-44494f7376fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:45:55.569454 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.569328 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-model-cache\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:45:55.569454 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.569392 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-tmp-dir\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:45:55.569454 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.569402 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-home\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:45:55.571205 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.571179 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5f65e30-e8d4-4991-ae51-44494f7376fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c5f65e30-e8d4-4991-ae51-44494f7376fb" (UID: "c5f65e30-e8d4-4991-ae51-44494f7376fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:45:55.571269 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.571212 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-dshm" (OuterVolumeSpecName: "dshm") pod "c5f65e30-e8d4-4991-ae51-44494f7376fb" (UID: "c5f65e30-e8d4-4991-ae51-44494f7376fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:45:55.571311 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.571264 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f65e30-e8d4-4991-ae51-44494f7376fb-kube-api-access-qptzz" (OuterVolumeSpecName: "kube-api-access-qptzz") pod "c5f65e30-e8d4-4991-ae51-44494f7376fb" (UID: "c5f65e30-e8d4-4991-ae51-44494f7376fb"). InnerVolumeSpecName "kube-api-access-qptzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:45:55.669833 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.669784 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qptzz\" (UniqueName: \"kubernetes.io/projected/c5f65e30-e8d4-4991-ae51-44494f7376fb-kube-api-access-qptzz\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:45:55.669833 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.669824 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f65e30-e8d4-4991-ae51-44494f7376fb-dshm\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:45:55.669833 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:55.669835 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f65e30-e8d4-4991-ae51-44494f7376fb-tls-certs\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:45:56.428395 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:56.428365 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" Jul 02 19:45:56.428872 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:56.428397 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj" event={"ID":"c5f65e30-e8d4-4991-ae51-44494f7376fb","Type":"ContainerDied","Data":"5fc1c2686a4e281884c63dd6f84127edb4bd0c5d2260d49795bf754ddc69fcf2"} Jul 02 19:45:56.428872 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:56.428447 2565 scope.go:117] "RemoveContainer" containerID="9e6eb682212328c13035725914e444600c390db1287247a0160c04fcb64b3c45" Jul 02 19:45:56.449744 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:56.449715 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj"] Jul 02 19:45:56.454050 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:56.454017 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0c31dc/auth-test-model-0c31dc-kserve-5cdc697fcf-9sbzj"] Jul 02 19:45:57.399726 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:57.399692 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d4a7075-b92b-4942-83c0-e664df1278c0" path="/var/lib/kubelet/pods/8d4a7075-b92b-4942-83c0-e664df1278c0/volumes" Jul 02 19:45:57.400079 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:45:57.400067 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5f65e30-e8d4-4991-ae51-44494f7376fb" path="/var/lib/kubelet/pods/c5f65e30-e8d4-4991-ae51-44494f7376fb/volumes" Jul 02 19:46:02.306315 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:02.306270 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:46:02.306745 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:02.306389 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:46:02.319234 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:02.319205 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:46:02.470332 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:02.470303 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:46:03.195540 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:03.195491 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:46:14.882541 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:14.882506 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs"] Jul 02 19:46:14.883010 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:14.882926 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d4a7075-b92b-4942-83c0-e664df1278c0" containerName="main" Jul 02 19:46:14.883010 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:14.882944 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d4a7075-b92b-4942-83c0-e664df1278c0" containerName="main" Jul 02 19:46:14.883010 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:14.882963 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5f65e30-e8d4-4991-ae51-44494f7376fb" containerName="main" Jul 02 19:46:14.883010 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:14.882970 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5f65e30-e8d4-4991-ae51-44494f7376fb" containerName="main" Jul 02 19:46:14.883212 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:14.883061 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5f65e30-e8d4-4991-ae51-44494f7376fb" containerName="main" Jul 02 19:46:14.883212 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:14.883079 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d4a7075-b92b-4942-83c0-e664df1278c0" containerName="main" Jul 02 19:46:14.887165 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:14.887143 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:14.890150 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:14.890127 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0c31dc\"/\"openshift-service-ca.crt\"" Jul 02 19:46:14.891378 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:14.891341 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0c31dc\"/\"sub-test-model-0c31dc-kserve-self-signed-certs\"" Jul 02 19:46:14.891494 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:14.891377 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0c31dc\"/\"default-dockercfg-g6cjj\"" Jul 02 19:46:14.891494 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:14.891386 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0c31dc\"/\"kube-root-ca.crt\"" Jul 02 19:46:14.896182 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:14.896155 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs"] Jul 02 19:46:14.943418 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:14.943339 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-home\") pod \"sub-test-model-0c31dc-kserve-55fb955659-ljqjs\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:14.943601 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:14.943456 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-tmp-dir\") pod \"sub-test-model-0c31dc-kserve-55fb955659-ljqjs\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:14.943601 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:14.943488 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-tls-certs\") pod \"sub-test-model-0c31dc-kserve-55fb955659-ljqjs\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:14.943601 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:14.943527 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-dshm\") pod \"sub-test-model-0c31dc-kserve-55fb955659-ljqjs\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:14.943729 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:14.943628 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-model-cache\") pod \"sub-test-model-0c31dc-kserve-55fb955659-ljqjs\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:14.943729 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:14.943666 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kxpr\" (UniqueName: \"kubernetes.io/projected/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-kube-api-access-2kxpr\") pod \"sub-test-model-0c31dc-kserve-55fb955659-ljqjs\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:15.044782 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:15.044735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-model-cache\") pod \"sub-test-model-0c31dc-kserve-55fb955659-ljqjs\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:15.044782 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:15.044788 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kxpr\" (UniqueName: \"kubernetes.io/projected/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-kube-api-access-2kxpr\") pod \"sub-test-model-0c31dc-kserve-55fb955659-ljqjs\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:15.045025 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:15.044824 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-home\") pod \"sub-test-model-0c31dc-kserve-55fb955659-ljqjs\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:15.045025 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:15.044859 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-tmp-dir\") pod \"sub-test-model-0c31dc-kserve-55fb955659-ljqjs\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:15.045025 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:15.044886 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-tls-certs\") pod \"sub-test-model-0c31dc-kserve-55fb955659-ljqjs\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:15.045025 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:15.044937 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-dshm\") pod \"sub-test-model-0c31dc-kserve-55fb955659-ljqjs\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:15.045230 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:15.045188 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-model-cache\") pod \"sub-test-model-0c31dc-kserve-55fb955659-ljqjs\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:15.045290 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:15.045264 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-tmp-dir\") pod \"sub-test-model-0c31dc-kserve-55fb955659-ljqjs\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:15.045332 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:15.045286 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-home\") pod \"sub-test-model-0c31dc-kserve-55fb955659-ljqjs\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:15.047211 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:15.047183 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-dshm\") pod \"sub-test-model-0c31dc-kserve-55fb955659-ljqjs\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:15.047612 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:15.047588 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-tls-certs\") pod \"sub-test-model-0c31dc-kserve-55fb955659-ljqjs\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:15.053013 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:15.052982 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kxpr\" (UniqueName: \"kubernetes.io/projected/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-kube-api-access-2kxpr\") pod \"sub-test-model-0c31dc-kserve-55fb955659-ljqjs\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:15.197380 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:15.197336 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:15.335624 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:15.335598 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs"] Jul 02 19:46:15.337531 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:46:15.337503 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52227e5b_23fc_4d8c_80c7_9ba7e4d3fe3e.slice/crio-23180c030f914a1ee21b4f8459d8bc9cb6ef6f0d2e21f50ef02d05b88fe86ed9 WatchSource:0}: Error finding container 23180c030f914a1ee21b4f8459d8bc9cb6ef6f0d2e21f50ef02d05b88fe86ed9: Status 404 returned error can't find the container with id 23180c030f914a1ee21b4f8459d8bc9cb6ef6f0d2e21f50ef02d05b88fe86ed9 Jul 02 19:46:15.516255 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:15.516154 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" event={"ID":"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e","Type":"ContainerStarted","Data":"3e614149e76c5df8ffe312b640351e2ac5062d41c9521b3c85a150693e8e625f"} Jul 02 19:46:15.516255 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:15.516208 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" event={"ID":"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e","Type":"ContainerStarted","Data":"23180c030f914a1ee21b4f8459d8bc9cb6ef6f0d2e21f50ef02d05b88fe86ed9"} Jul 02 19:46:15.533784 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:15.533710 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" podStartSLOduration=1.533693175 podStartE2EDuration="1.533693175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:46:15.532852781 +0000 UTC m=+2314.648204005" watchObservedRunningTime="2026-07-02 19:46:15.533693175 +0000 UTC m=+2314.649044426" Jul 02 19:46:19.509442 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:19.509403 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:46:25.198543 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:25.198507 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:25.198543 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:25.198554 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:25.211260 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:25.211227 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:25.572512 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:25.572424 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:46:38.805083 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:38.804993 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:46:56.595744 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:46:56.595702 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:47:02.416866 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:02.416828 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x"] Jul 02 19:47:03.407211 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:03.407177 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs"] Jul 02 19:47:10.104243 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.104205 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm"] Jul 02 19:47:10.108173 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.108145 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.110808 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.110779 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-67cf22\"/\"rate-test-model-67cf22-kserve-self-signed-certs\"" Jul 02 19:47:10.117936 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.117904 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm"] Jul 02 19:47:10.230860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.230823 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-dshm\") pod \"rate-test-model-67cf22-kserve-75564ffd4c-hqgcm\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.231067 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.230871 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-model-cache\") pod \"rate-test-model-67cf22-kserve-75564ffd4c-hqgcm\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.231067 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.230909 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-home\") pod \"rate-test-model-67cf22-kserve-75564ffd4c-hqgcm\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.231067 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.230954 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6140c4-a4b4-4a38-9303-aeb4072c24af-tls-certs\") pod \"rate-test-model-67cf22-kserve-75564ffd4c-hqgcm\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.231067 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.231046 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-tmp-dir\") pod \"rate-test-model-67cf22-kserve-75564ffd4c-hqgcm\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.231238 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.231086 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqhft\" (UniqueName: \"kubernetes.io/projected/3e6140c4-a4b4-4a38-9303-aeb4072c24af-kube-api-access-kqhft\") pod \"rate-test-model-67cf22-kserve-75564ffd4c-hqgcm\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.332314 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.332266 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-home\") pod \"rate-test-model-67cf22-kserve-75564ffd4c-hqgcm\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.332536 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.332332 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6140c4-a4b4-4a38-9303-aeb4072c24af-tls-certs\") pod \"rate-test-model-67cf22-kserve-75564ffd4c-hqgcm\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.332536 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.332424 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-tmp-dir\") pod \"rate-test-model-67cf22-kserve-75564ffd4c-hqgcm\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.332536 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.332456 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqhft\" (UniqueName: \"kubernetes.io/projected/3e6140c4-a4b4-4a38-9303-aeb4072c24af-kube-api-access-kqhft\") pod \"rate-test-model-67cf22-kserve-75564ffd4c-hqgcm\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.332536 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.332494 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-dshm\") pod \"rate-test-model-67cf22-kserve-75564ffd4c-hqgcm\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.332536 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.332522 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-model-cache\") pod \"rate-test-model-67cf22-kserve-75564ffd4c-hqgcm\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.332804 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.332740 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-home\") pod \"rate-test-model-67cf22-kserve-75564ffd4c-hqgcm\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.332804 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.332790 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-tmp-dir\") pod \"rate-test-model-67cf22-kserve-75564ffd4c-hqgcm\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.332959 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.332853 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-model-cache\") pod \"rate-test-model-67cf22-kserve-75564ffd4c-hqgcm\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.334647 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.334625 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-dshm\") pod \"rate-test-model-67cf22-kserve-75564ffd4c-hqgcm\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.334812 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.334797 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6140c4-a4b4-4a38-9303-aeb4072c24af-tls-certs\") pod \"rate-test-model-67cf22-kserve-75564ffd4c-hqgcm\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.341218 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.341194 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqhft\" (UniqueName: \"kubernetes.io/projected/3e6140c4-a4b4-4a38-9303-aeb4072c24af-kube-api-access-kqhft\") pod \"rate-test-model-67cf22-kserve-75564ffd4c-hqgcm\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.418083 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.418040 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:10.554207 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.554183 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm"] Jul 02 19:47:10.556298 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:47:10.556268 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e6140c4_a4b4_4a38_9303_aeb4072c24af.slice/crio-c3e4d33436e0e9abd3d36efb49d1adb1168353525972f76bcbb869ac1a082218 WatchSource:0}: Error finding container c3e4d33436e0e9abd3d36efb49d1adb1168353525972f76bcbb869ac1a082218: Status 404 returned error can't find the container with id c3e4d33436e0e9abd3d36efb49d1adb1168353525972f76bcbb869ac1a082218 Jul 02 19:47:10.763196 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.763099 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" event={"ID":"3e6140c4-a4b4-4a38-9303-aeb4072c24af","Type":"ContainerStarted","Data":"e9663532197adda15122f3e1987db29b0d55ab0f273f5e19edc1358c7c99471a"} Jul 02 19:47:10.763196 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.763154 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" event={"ID":"3e6140c4-a4b4-4a38-9303-aeb4072c24af","Type":"ContainerStarted","Data":"c3e4d33436e0e9abd3d36efb49d1adb1168353525972f76bcbb869ac1a082218"} Jul 02 19:47:10.781875 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:10.781824 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" podStartSLOduration=0.781807434 podStartE2EDuration="781.807434ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:47:10.779775335 +0000 UTC m=+2369.895126559" watchObservedRunningTime="2026-07-02 19:47:10.781807434 +0000 UTC m=+2369.897158657" Jul 02 19:47:13.097855 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:13.097760 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:47:17.436634 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.436586 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" podUID="c7d6337a-b571-4960-82ee-935cef24d4dc" containerName="main" containerID="cri-o://f38aa4722649b17cabc2e19ae324293496fd0f7666ce3b157d9ea763adb8cdeb" gracePeriod=45 Jul 02 19:47:17.691268 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.691197 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:47:17.794479 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.794442 2565 generic.go:356] "Generic (PLEG): container finished" podID="c7d6337a-b571-4960-82ee-935cef24d4dc" containerID="f38aa4722649b17cabc2e19ae324293496fd0f7666ce3b157d9ea763adb8cdeb" exitCode=0 Jul 02 19:47:17.794773 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.794485 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" Jul 02 19:47:17.794773 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.794529 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" event={"ID":"c7d6337a-b571-4960-82ee-935cef24d4dc","Type":"ContainerDied","Data":"f38aa4722649b17cabc2e19ae324293496fd0f7666ce3b157d9ea763adb8cdeb"} Jul 02 19:47:17.794773 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.794568 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x" event={"ID":"c7d6337a-b571-4960-82ee-935cef24d4dc","Type":"ContainerDied","Data":"e6072bba9d02b740f2cc6e75db14387f84ef4a9017278321a7770e659b698b9a"} Jul 02 19:47:17.794773 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.794585 2565 scope.go:117] "RemoveContainer" containerID="f38aa4722649b17cabc2e19ae324293496fd0f7666ce3b157d9ea763adb8cdeb" Jul 02 19:47:17.798532 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.798513 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-model-cache\") pod \"c7d6337a-b571-4960-82ee-935cef24d4dc\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " Jul 02 19:47:17.798644 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.798543 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-dshm\") pod \"c7d6337a-b571-4960-82ee-935cef24d4dc\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " Jul 02 19:47:17.798644 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.798566 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-home\") pod \"c7d6337a-b571-4960-82ee-935cef24d4dc\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " Jul 02 19:47:17.798644 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.798602 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8qzh\" (UniqueName: \"kubernetes.io/projected/c7d6337a-b571-4960-82ee-935cef24d4dc-kube-api-access-x8qzh\") pod \"c7d6337a-b571-4960-82ee-935cef24d4dc\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " Jul 02 19:47:17.798801 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.798643 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-tmp-dir\") pod \"c7d6337a-b571-4960-82ee-935cef24d4dc\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " Jul 02 19:47:17.798801 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.798706 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d6337a-b571-4960-82ee-935cef24d4dc-tls-certs\") pod \"c7d6337a-b571-4960-82ee-935cef24d4dc\" (UID: \"c7d6337a-b571-4960-82ee-935cef24d4dc\") " Jul 02 19:47:17.798801 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.798773 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7d6337a-b571-4960-82ee-935cef24d4dc" (UID: "c7d6337a-b571-4960-82ee-935cef24d4dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:47:17.798947 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.798880 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-home" (OuterVolumeSpecName: "home") pod "c7d6337a-b571-4960-82ee-935cef24d4dc" (UID: "c7d6337a-b571-4960-82ee-935cef24d4dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:47:17.799014 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.798967 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7d6337a-b571-4960-82ee-935cef24d4dc" (UID: "c7d6337a-b571-4960-82ee-935cef24d4dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:47:17.799072 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.799029 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-model-cache\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:47:17.799072 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.799060 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-home\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:47:17.800740 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.800714 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-dshm" (OuterVolumeSpecName: "dshm") pod "c7d6337a-b571-4960-82ee-935cef24d4dc" (UID: "c7d6337a-b571-4960-82ee-935cef24d4dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:47:17.800832 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.800718 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d6337a-b571-4960-82ee-935cef24d4dc-kube-api-access-x8qzh" (OuterVolumeSpecName: "kube-api-access-x8qzh") pod "c7d6337a-b571-4960-82ee-935cef24d4dc" (UID: "c7d6337a-b571-4960-82ee-935cef24d4dc"). InnerVolumeSpecName "kube-api-access-x8qzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:47:17.800832 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.800764 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d6337a-b571-4960-82ee-935cef24d4dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7d6337a-b571-4960-82ee-935cef24d4dc" (UID: "c7d6337a-b571-4960-82ee-935cef24d4dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:47:17.805789 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.805591 2565 scope.go:117] "RemoveContainer" containerID="f38aa4722649b17cabc2e19ae324293496fd0f7666ce3b157d9ea763adb8cdeb" Jul 02 19:47:17.806096 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:47:17.806020 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f38aa4722649b17cabc2e19ae324293496fd0f7666ce3b157d9ea763adb8cdeb\": container with ID starting with f38aa4722649b17cabc2e19ae324293496fd0f7666ce3b157d9ea763adb8cdeb not found: ID does not exist" containerID="f38aa4722649b17cabc2e19ae324293496fd0f7666ce3b157d9ea763adb8cdeb" Jul 02 19:47:17.806096 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.806057 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f38aa4722649b17cabc2e19ae324293496fd0f7666ce3b157d9ea763adb8cdeb"} err="failed to get container status \"f38aa4722649b17cabc2e19ae324293496fd0f7666ce3b157d9ea763adb8cdeb\": rpc error: code = NotFound desc = could not find container \"f38aa4722649b17cabc2e19ae324293496fd0f7666ce3b157d9ea763adb8cdeb\": container with ID starting with f38aa4722649b17cabc2e19ae324293496fd0f7666ce3b157d9ea763adb8cdeb not found: ID does not exist" Jul 02 19:47:17.899705 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.899667 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d6337a-b571-4960-82ee-935cef24d4dc-tls-certs\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:47:17.899705 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.899698 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-dshm\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:47:17.899705 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.899708 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8qzh\" (UniqueName: \"kubernetes.io/projected/c7d6337a-b571-4960-82ee-935cef24d4dc-kube-api-access-x8qzh\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:47:17.899957 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:17.899720 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d6337a-b571-4960-82ee-935cef24d4dc-tmp-dir\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:47:18.115405 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.115363 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x"] Jul 02 19:47:18.119723 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.119692 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-67cf22/sub-test-model-67cf22-kserve-6645d9bb4c-c9d2x"] Jul 02 19:47:18.426957 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.426889 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" podUID="52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e" containerName="main" containerID="cri-o://3e614149e76c5df8ffe312b640351e2ac5062d41c9521b3c85a150693e8e625f" gracePeriod=45 Jul 02 19:47:18.678330 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.678267 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:47:18.801258 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.801225 2565 generic.go:356] "Generic (PLEG): container finished" podID="52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e" containerID="3e614149e76c5df8ffe312b640351e2ac5062d41c9521b3c85a150693e8e625f" exitCode=0 Jul 02 19:47:18.801437 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.801267 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" Jul 02 19:47:18.801437 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.801323 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" event={"ID":"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e","Type":"ContainerDied","Data":"3e614149e76c5df8ffe312b640351e2ac5062d41c9521b3c85a150693e8e625f"} Jul 02 19:47:18.801437 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.801378 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs" event={"ID":"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e","Type":"ContainerDied","Data":"23180c030f914a1ee21b4f8459d8bc9cb6ef6f0d2e21f50ef02d05b88fe86ed9"} Jul 02 19:47:18.801437 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.801399 2565 scope.go:117] "RemoveContainer" containerID="3e614149e76c5df8ffe312b640351e2ac5062d41c9521b3c85a150693e8e625f" Jul 02 19:47:18.809412 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.809390 2565 scope.go:117] "RemoveContainer" containerID="3e614149e76c5df8ffe312b640351e2ac5062d41c9521b3c85a150693e8e625f" Jul 02 19:47:18.809657 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:47:18.809638 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e614149e76c5df8ffe312b640351e2ac5062d41c9521b3c85a150693e8e625f\": container with ID starting with 3e614149e76c5df8ffe312b640351e2ac5062d41c9521b3c85a150693e8e625f not found: ID does not exist" containerID="3e614149e76c5df8ffe312b640351e2ac5062d41c9521b3c85a150693e8e625f" Jul 02 19:47:18.809710 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.809668 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e614149e76c5df8ffe312b640351e2ac5062d41c9521b3c85a150693e8e625f"} err="failed to get container status \"3e614149e76c5df8ffe312b640351e2ac5062d41c9521b3c85a150693e8e625f\": rpc error: code = NotFound desc = could not find container \"3e614149e76c5df8ffe312b640351e2ac5062d41c9521b3c85a150693e8e625f\": container with ID starting with 3e614149e76c5df8ffe312b640351e2ac5062d41c9521b3c85a150693e8e625f not found: ID does not exist" Jul 02 19:47:18.812020 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.812003 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-home\") pod \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " Jul 02 19:47:18.812089 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.812030 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-dshm\") pod \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " Jul 02 19:47:18.812089 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.812064 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-tmp-dir\") pod \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " Jul 02 19:47:18.812089 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.812082 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kxpr\" (UniqueName: \"kubernetes.io/projected/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-kube-api-access-2kxpr\") pod \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " Jul 02 19:47:18.812203 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.812127 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-tls-certs\") pod \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " Jul 02 19:47:18.812203 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.812163 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-model-cache\") pod \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\" (UID: \"52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e\") " Jul 02 19:47:18.812309 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.812279 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-home" (OuterVolumeSpecName: "home") pod "52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e" (UID: "52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:47:18.812309 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.812288 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e" (UID: "52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:47:18.812506 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.812481 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-model-cache" (OuterVolumeSpecName: "model-cache") pod "52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e" (UID: "52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:47:18.812506 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.812490 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-home\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:47:18.812650 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.812523 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-tmp-dir\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:47:18.814289 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.814265 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e" (UID: "52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:47:18.814620 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.814600 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-dshm" (OuterVolumeSpecName: "dshm") pod "52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e" (UID: "52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:47:18.814620 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.814611 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-kube-api-access-2kxpr" (OuterVolumeSpecName: "kube-api-access-2kxpr") pod "52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e" (UID: "52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e"). InnerVolumeSpecName "kube-api-access-2kxpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:47:18.913175 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.913135 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-tls-certs\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:47:18.913175 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.913168 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-model-cache\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:47:18.913175 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.913180 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-dshm\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:47:18.913458 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:18.913188 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kxpr\" (UniqueName: \"kubernetes.io/projected/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e-kube-api-access-2kxpr\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:47:19.122831 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:19.122798 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs"] Jul 02 19:47:19.132441 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:19.132408 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0c31dc/sub-test-model-0c31dc-kserve-55fb955659-ljqjs"] Jul 02 19:47:19.399833 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:19.399806 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e" path="/var/lib/kubelet/pods/52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e/volumes" Jul 02 19:47:19.400199 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:19.400186 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7d6337a-b571-4960-82ee-935cef24d4dc" path="/var/lib/kubelet/pods/c7d6337a-b571-4960-82ee-935cef24d4dc/volumes" Jul 02 19:47:20.418997 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:20.418944 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:20.418997 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:20.419009 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:20.431689 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:20.431662 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:20.822821 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:20.822732 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:47:41.483479 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:41.483450 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:47:41.485655 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:41.485631 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:47:43.878753 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:43.878717 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2"] Jul 02 19:47:43.879129 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:43.879007 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d6337a-b571-4960-82ee-935cef24d4dc" containerName="main" Jul 02 19:47:43.879129 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:43.879018 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d6337a-b571-4960-82ee-935cef24d4dc" containerName="main" Jul 02 19:47:43.879129 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:43.879040 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e" containerName="main" Jul 02 19:47:43.879129 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:43.879045 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e" containerName="main" Jul 02 19:47:43.879129 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:43.879095 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d6337a-b571-4960-82ee-935cef24d4dc" containerName="main" Jul 02 19:47:43.879129 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:43.879105 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="52227e5b-23fc-4d8c-80c7-9ba7e4d3fe3e" containerName="main" Jul 02 19:47:43.883599 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:43.883577 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:43.887259 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:43.887217 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0c31dc\"/\"openshift-service-ca.crt\"" Jul 02 19:47:43.887259 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:43.887248 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0c31dc\"/\"default-dockercfg-g6cjj\"" Jul 02 19:47:43.887489 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:43.887229 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0c31dc\"/\"rate-test-model-0c31dc-kserve-self-signed-certs\"" Jul 02 19:47:43.887489 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:43.887271 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0c31dc\"/\"kube-root-ca.crt\"" Jul 02 19:47:43.892492 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:43.892465 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2"] Jul 02 19:47:43.923243 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:43.923205 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slfp4\" (UniqueName: \"kubernetes.io/projected/20c13c3b-f6fd-4573-9204-a6e96de5178f-kube-api-access-slfp4\") pod \"rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:43.923243 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:43.923242 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-model-cache\") pod \"rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:43.923538 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:43.923266 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20c13c3b-f6fd-4573-9204-a6e96de5178f-tls-certs\") pod \"rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:43.923538 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:43.923383 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-tmp-dir\") pod \"rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:43.923538 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:43.923464 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-home\") pod \"rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:43.923538 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:43.923479 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-dshm\") pod \"rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:44.024592 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:44.024536 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20c13c3b-f6fd-4573-9204-a6e96de5178f-tls-certs\") pod \"rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:44.024802 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:44.024620 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-tmp-dir\") pod \"rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:44.024802 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:44.024674 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-home\") pod \"rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:44.024802 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:44.024696 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-dshm\") pod \"rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:44.024802 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:44.024742 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slfp4\" (UniqueName: \"kubernetes.io/projected/20c13c3b-f6fd-4573-9204-a6e96de5178f-kube-api-access-slfp4\") pod \"rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:44.024802 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:44.024766 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-model-cache\") pod \"rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:44.025091 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:44.025070 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-model-cache\") pod \"rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:44.025091 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:44.025085 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-tmp-dir\") pod \"rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:44.025202 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:44.025150 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-home\") pod \"rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:44.027101 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:44.027064 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-dshm\") pod \"rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:44.027291 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:44.027263 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20c13c3b-f6fd-4573-9204-a6e96de5178f-tls-certs\") pod \"rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:44.040703 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:44.040673 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slfp4\" (UniqueName: \"kubernetes.io/projected/20c13c3b-f6fd-4573-9204-a6e96de5178f-kube-api-access-slfp4\") pod \"rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:44.193704 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:44.193669 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:44.329654 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:44.329630 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2"] Jul 02 19:47:44.331541 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:47:44.331517 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20c13c3b_f6fd_4573_9204_a6e96de5178f.slice/crio-469d945f6c33179d9cb385b9b61009c69af631ab6f173ccafef74355840bcd59 WatchSource:0}: Error finding container 469d945f6c33179d9cb385b9b61009c69af631ab6f173ccafef74355840bcd59: Status 404 returned error can't find the container with id 469d945f6c33179d9cb385b9b61009c69af631ab6f173ccafef74355840bcd59 Jul 02 19:47:44.921639 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:44.921603 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" event={"ID":"20c13c3b-f6fd-4573-9204-a6e96de5178f","Type":"ContainerStarted","Data":"75655b862b0d1e02aecd5e9e119ac41c70f64313c802847422b6915d6dc862c3"} Jul 02 19:47:44.922034 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:44.921648 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" event={"ID":"20c13c3b-f6fd-4573-9204-a6e96de5178f","Type":"ContainerStarted","Data":"469d945f6c33179d9cb385b9b61009c69af631ab6f173ccafef74355840bcd59"} Jul 02 19:47:44.942944 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:44.942886 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" podStartSLOduration=1.942870916 podStartE2EDuration="1.942870916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:47:44.940642427 +0000 UTC m=+2404.055993652" watchObservedRunningTime="2026-07-02 19:47:44.942870916 +0000 UTC m=+2404.058222139" Jul 02 19:47:47.698282 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:47.698239 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:47:54.194105 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:54.194061 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:54.194548 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:54.194133 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:54.206797 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:54.206770 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:47:54.978471 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:47:54.978441 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:48:01.704716 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:01.704670 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:48:16.399315 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:16.399277 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:48:22.987004 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:22.986965 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm"] Jul 02 19:48:23.850825 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:23.850785 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2"] Jul 02 19:48:25.893576 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:25.893533 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4"] Jul 02 19:48:25.895732 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:25.895696 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-67cf22-29717025-qf6g4"] Jul 02 19:48:27.400595 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:27.400554 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8af756a-e839-4346-95f6-145990b3a6f8" path="/var/lib/kubelet/pods/e8af756a-e839-4346-95f6-145990b3a6f8/volumes" Jul 02 19:48:29.991440 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:29.991405 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:48:38.006794 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.006748 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" podUID="3e6140c4-a4b4-4a38-9303-aeb4072c24af" containerName="main" containerID="cri-o://e9663532197adda15122f3e1987db29b0d55ab0f273f5e19edc1358c7c99471a" gracePeriod=45 Jul 02 19:48:38.162843 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.162811 2565 generic.go:356] "Generic (PLEG): container finished" podID="3e6140c4-a4b4-4a38-9303-aeb4072c24af" containerID="e9663532197adda15122f3e1987db29b0d55ab0f273f5e19edc1358c7c99471a" exitCode=0 Jul 02 19:48:38.163005 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.162882 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" event={"ID":"3e6140c4-a4b4-4a38-9303-aeb4072c24af","Type":"ContainerDied","Data":"e9663532197adda15122f3e1987db29b0d55ab0f273f5e19edc1358c7c99471a"} Jul 02 19:48:38.269666 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.269594 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:48:38.299168 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.299135 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-model-cache\") pod \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " Jul 02 19:48:38.299168 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.299174 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-tmp-dir\") pod \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " Jul 02 19:48:38.299445 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.299206 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-home\") pod \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " Jul 02 19:48:38.299445 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.299232 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6140c4-a4b4-4a38-9303-aeb4072c24af-tls-certs\") pod \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " Jul 02 19:48:38.299445 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.299250 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqhft\" (UniqueName: \"kubernetes.io/projected/3e6140c4-a4b4-4a38-9303-aeb4072c24af-kube-api-access-kqhft\") pod \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " Jul 02 19:48:38.299445 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.299309 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-dshm\") pod \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\" (UID: \"3e6140c4-a4b4-4a38-9303-aeb4072c24af\") " Jul 02 19:48:38.299660 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.299504 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e6140c4-a4b4-4a38-9303-aeb4072c24af" (UID: "3e6140c4-a4b4-4a38-9303-aeb4072c24af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:48:38.299660 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.299521 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e6140c4-a4b4-4a38-9303-aeb4072c24af" (UID: "3e6140c4-a4b4-4a38-9303-aeb4072c24af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:48:38.299660 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.299613 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-model-cache\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:48:38.299660 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.299630 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-tmp-dir\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:48:38.299660 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.299627 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-home" (OuterVolumeSpecName: "home") pod "3e6140c4-a4b4-4a38-9303-aeb4072c24af" (UID: "3e6140c4-a4b4-4a38-9303-aeb4072c24af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:48:38.301491 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.301453 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e6140c4-a4b4-4a38-9303-aeb4072c24af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e6140c4-a4b4-4a38-9303-aeb4072c24af" (UID: "3e6140c4-a4b4-4a38-9303-aeb4072c24af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:48:38.301621 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.301532 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6140c4-a4b4-4a38-9303-aeb4072c24af-kube-api-access-kqhft" (OuterVolumeSpecName: "kube-api-access-kqhft") pod "3e6140c4-a4b4-4a38-9303-aeb4072c24af" (UID: "3e6140c4-a4b4-4a38-9303-aeb4072c24af"). InnerVolumeSpecName "kube-api-access-kqhft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:48:38.301818 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.301796 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-dshm" (OuterVolumeSpecName: "dshm") pod "3e6140c4-a4b4-4a38-9303-aeb4072c24af" (UID: "3e6140c4-a4b4-4a38-9303-aeb4072c24af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:48:38.400066 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.400019 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-home\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:48:38.400066 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.400056 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e6140c4-a4b4-4a38-9303-aeb4072c24af-tls-certs\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:48:38.400066 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.400067 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqhft\" (UniqueName: \"kubernetes.io/projected/3e6140c4-a4b4-4a38-9303-aeb4072c24af-kube-api-access-kqhft\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:48:38.400066 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.400075 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e6140c4-a4b4-4a38-9303-aeb4072c24af-dshm\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:48:38.870486 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:38.870430 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" podUID="20c13c3b-f6fd-4573-9204-a6e96de5178f" containerName="main" containerID="cri-o://75655b862b0d1e02aecd5e9e119ac41c70f64313c802847422b6915d6dc862c3" gracePeriod=45 Jul 02 19:48:39.130229 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.130162 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:48:39.169984 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.169954 2565 generic.go:356] "Generic (PLEG): container finished" podID="20c13c3b-f6fd-4573-9204-a6e96de5178f" containerID="75655b862b0d1e02aecd5e9e119ac41c70f64313c802847422b6915d6dc862c3" exitCode=0 Jul 02 19:48:39.170182 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.169999 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" Jul 02 19:48:39.170182 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.170042 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" event={"ID":"20c13c3b-f6fd-4573-9204-a6e96de5178f","Type":"ContainerDied","Data":"75655b862b0d1e02aecd5e9e119ac41c70f64313c802847422b6915d6dc862c3"} Jul 02 19:48:39.170182 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.170085 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2" event={"ID":"20c13c3b-f6fd-4573-9204-a6e96de5178f","Type":"ContainerDied","Data":"469d945f6c33179d9cb385b9b61009c69af631ab6f173ccafef74355840bcd59"} Jul 02 19:48:39.170182 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.170110 2565 scope.go:117] "RemoveContainer" containerID="75655b862b0d1e02aecd5e9e119ac41c70f64313c802847422b6915d6dc862c3" Jul 02 19:48:39.171435 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.171410 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" event={"ID":"3e6140c4-a4b4-4a38-9303-aeb4072c24af","Type":"ContainerDied","Data":"c3e4d33436e0e9abd3d36efb49d1adb1168353525972f76bcbb869ac1a082218"} Jul 02 19:48:39.171550 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.171452 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm" Jul 02 19:48:39.181293 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.181275 2565 scope.go:117] "RemoveContainer" containerID="75655b862b0d1e02aecd5e9e119ac41c70f64313c802847422b6915d6dc862c3" Jul 02 19:48:39.181627 ip-10-0-142-165 kubenswrapper[2565]: E0702 19:48:39.181603 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75655b862b0d1e02aecd5e9e119ac41c70f64313c802847422b6915d6dc862c3\": container with ID starting with 75655b862b0d1e02aecd5e9e119ac41c70f64313c802847422b6915d6dc862c3 not found: ID does not exist" containerID="75655b862b0d1e02aecd5e9e119ac41c70f64313c802847422b6915d6dc862c3" Jul 02 19:48:39.181734 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.181632 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75655b862b0d1e02aecd5e9e119ac41c70f64313c802847422b6915d6dc862c3"} err="failed to get container status \"75655b862b0d1e02aecd5e9e119ac41c70f64313c802847422b6915d6dc862c3\": rpc error: code = NotFound desc = could not find container \"75655b862b0d1e02aecd5e9e119ac41c70f64313c802847422b6915d6dc862c3\": container with ID starting with 75655b862b0d1e02aecd5e9e119ac41c70f64313c802847422b6915d6dc862c3 not found: ID does not exist" Jul 02 19:48:39.181734 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.181656 2565 scope.go:117] "RemoveContainer" containerID="e9663532197adda15122f3e1987db29b0d55ab0f273f5e19edc1358c7c99471a" Jul 02 19:48:39.204536 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.204503 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm"] Jul 02 19:48:39.206117 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.206095 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slfp4\" (UniqueName: \"kubernetes.io/projected/20c13c3b-f6fd-4573-9204-a6e96de5178f-kube-api-access-slfp4\") pod \"20c13c3b-f6fd-4573-9204-a6e96de5178f\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " Jul 02 19:48:39.206179 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.206128 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-tmp-dir\") pod \"20c13c3b-f6fd-4573-9204-a6e96de5178f\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " Jul 02 19:48:39.206179 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.206146 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-home\") pod \"20c13c3b-f6fd-4573-9204-a6e96de5178f\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " Jul 02 19:48:39.206179 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.206168 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20c13c3b-f6fd-4573-9204-a6e96de5178f-tls-certs\") pod \"20c13c3b-f6fd-4573-9204-a6e96de5178f\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " Jul 02 19:48:39.206272 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.206233 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-dshm\") pod \"20c13c3b-f6fd-4573-9204-a6e96de5178f\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " Jul 02 19:48:39.206308 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.206295 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-model-cache\") pod \"20c13c3b-f6fd-4573-9204-a6e96de5178f\" (UID: \"20c13c3b-f6fd-4573-9204-a6e96de5178f\") " Jul 02 19:48:39.206464 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.206437 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-home" (OuterVolumeSpecName: "home") pod "20c13c3b-f6fd-4573-9204-a6e96de5178f" (UID: "20c13c3b-f6fd-4573-9204-a6e96de5178f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:48:39.206464 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.206441 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20c13c3b-f6fd-4573-9204-a6e96de5178f" (UID: "20c13c3b-f6fd-4573-9204-a6e96de5178f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:48:39.206645 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.206593 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-tmp-dir\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:48:39.206645 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.206614 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-home\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:48:39.206645 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.206625 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-model-cache" (OuterVolumeSpecName: "model-cache") pod "20c13c3b-f6fd-4573-9204-a6e96de5178f" (UID: "20c13c3b-f6fd-4573-9204-a6e96de5178f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:48:39.208355 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.208319 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20c13c3b-f6fd-4573-9204-a6e96de5178f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20c13c3b-f6fd-4573-9204-a6e96de5178f" (UID: "20c13c3b-f6fd-4573-9204-a6e96de5178f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:48:39.208461 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.208325 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-dshm" (OuterVolumeSpecName: "dshm") pod "20c13c3b-f6fd-4573-9204-a6e96de5178f" (UID: "20c13c3b-f6fd-4573-9204-a6e96de5178f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:48:39.208461 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.208411 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c13c3b-f6fd-4573-9204-a6e96de5178f-kube-api-access-slfp4" (OuterVolumeSpecName: "kube-api-access-slfp4") pod "20c13c3b-f6fd-4573-9204-a6e96de5178f" (UID: "20c13c3b-f6fd-4573-9204-a6e96de5178f"). InnerVolumeSpecName "kube-api-access-slfp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:48:39.212595 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.212569 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-67cf22/rate-test-model-67cf22-kserve-75564ffd4c-hqgcm"] Jul 02 19:48:39.307333 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.307294 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-dshm\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:48:39.307333 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.307329 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20c13c3b-f6fd-4573-9204-a6e96de5178f-model-cache\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:48:39.307598 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.307363 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slfp4\" (UniqueName: \"kubernetes.io/projected/20c13c3b-f6fd-4573-9204-a6e96de5178f-kube-api-access-slfp4\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:48:39.307598 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.307379 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20c13c3b-f6fd-4573-9204-a6e96de5178f-tls-certs\") on node \"ip-10-0-142-165.ec2.internal\" DevicePath \"\"" Jul 02 19:48:39.399831 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.399807 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e6140c4-a4b4-4a38-9303-aeb4072c24af" path="/var/lib/kubelet/pods/3e6140c4-a4b4-4a38-9303-aeb4072c24af/volumes" Jul 02 19:48:39.486541 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.486507 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2"] Jul 02 19:48:39.491469 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:39.491440 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0c31dc/rate-test-model-0c31dc-kserve-769cf9bb7f-8rlm2"] Jul 02 19:48:41.399658 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:41.399625 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20c13c3b-f6fd-4573-9204-a6e96de5178f" path="/var/lib/kubelet/pods/20c13c3b-f6fd-4573-9204-a6e96de5178f/volumes" Jul 02 19:48:55.288360 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:55.288307 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h"] Jul 02 19:48:55.291327 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:55.291305 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-0c31dc-29717025-pj86h"] Jul 02 19:48:55.400161 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:55.400122 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f999c236-0aa1-44f7-b04f-a0bb33ab1a15" path="/var/lib/kubelet/pods/f999c236-0aa1-44f7-b04f-a0bb33ab1a15/volumes" Jul 02 19:48:58.197926 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:48:58.197890 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-s5gfq"] Jul 02 19:49:08.146644 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:08.146614 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d8ddf44b5-cw5hx_d7ae933f-25af-4598-8fd8-4cb71a8ac1d1/copy-manifests/0.log" Jul 02 19:49:08.153517 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:08.153490 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d8ddf44b5-cw5hx_d7ae933f-25af-4598-8fd8-4cb71a8ac1d1/manager/0.log" Jul 02 19:49:10.589149 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:10.589120 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-s5gfq_2a9bc9fe-c25f-40af-b54c-b9ff2d91b22b/limitador/0.log" Jul 02 19:49:11.439332 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:11.439298 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bdf57ddb6-5k2qt_ef604999-a8e7-41ac-88ef-c5465fe39c9d/kube-auth-proxy/0.log" Jul 02 19:49:12.858928 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:12.858898 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-gsz4s_68722bde-8b53-4e6b-81e8-96044fb5de1d/main/0.log" Jul 02 19:49:16.622187 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:16.622153 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ll6bw/must-gather-f9z84"] Jul 02 19:49:16.622593 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:16.622472 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e6140c4-a4b4-4a38-9303-aeb4072c24af" containerName="main" Jul 02 19:49:16.622593 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:16.622484 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e6140c4-a4b4-4a38-9303-aeb4072c24af" containerName="main" Jul 02 19:49:16.622593 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:16.622493 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20c13c3b-f6fd-4573-9204-a6e96de5178f" containerName="main" Jul 02 19:49:16.622593 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:16.622498 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="20c13c3b-f6fd-4573-9204-a6e96de5178f" containerName="main" Jul 02 19:49:16.622593 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:16.622567 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e6140c4-a4b4-4a38-9303-aeb4072c24af" containerName="main" Jul 02 19:49:16.622593 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:16.622574 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="20c13c3b-f6fd-4573-9204-a6e96de5178f" containerName="main" Jul 02 19:49:16.625532 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:16.625509 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ll6bw/must-gather-f9z84" Jul 02 19:49:16.628010 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:16.627990 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ll6bw\"/\"kube-root-ca.crt\"" Jul 02 19:49:16.629109 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:16.629091 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ll6bw\"/\"openshift-service-ca.crt\"" Jul 02 19:49:16.629203 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:16.629117 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ll6bw\"/\"default-dockercfg-9shf6\"" Jul 02 19:49:16.635998 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:16.635968 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ll6bw/must-gather-f9z84"] Jul 02 19:49:16.675930 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:16.675897 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5gkp\" (UniqueName: \"kubernetes.io/projected/baa0f01d-a3e5-4bfe-bead-7ba6ecb45084-kube-api-access-t5gkp\") pod \"must-gather-f9z84\" (UID: \"baa0f01d-a3e5-4bfe-bead-7ba6ecb45084\") " pod="openshift-must-gather-ll6bw/must-gather-f9z84" Jul 02 19:49:16.675930 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:16.675942 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/baa0f01d-a3e5-4bfe-bead-7ba6ecb45084-must-gather-output\") pod \"must-gather-f9z84\" (UID: \"baa0f01d-a3e5-4bfe-bead-7ba6ecb45084\") " pod="openshift-must-gather-ll6bw/must-gather-f9z84" Jul 02 19:49:16.776363 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:16.776314 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5gkp\" (UniqueName: \"kubernetes.io/projected/baa0f01d-a3e5-4bfe-bead-7ba6ecb45084-kube-api-access-t5gkp\") pod \"must-gather-f9z84\" (UID: \"baa0f01d-a3e5-4bfe-bead-7ba6ecb45084\") " pod="openshift-must-gather-ll6bw/must-gather-f9z84" Jul 02 19:49:16.776556 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:16.776417 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/baa0f01d-a3e5-4bfe-bead-7ba6ecb45084-must-gather-output\") pod \"must-gather-f9z84\" (UID: \"baa0f01d-a3e5-4bfe-bead-7ba6ecb45084\") " pod="openshift-must-gather-ll6bw/must-gather-f9z84" Jul 02 19:49:16.776770 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:16.776751 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/baa0f01d-a3e5-4bfe-bead-7ba6ecb45084-must-gather-output\") pod \"must-gather-f9z84\" (UID: \"baa0f01d-a3e5-4bfe-bead-7ba6ecb45084\") " pod="openshift-must-gather-ll6bw/must-gather-f9z84" Jul 02 19:49:16.787744 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:16.787720 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5gkp\" (UniqueName: \"kubernetes.io/projected/baa0f01d-a3e5-4bfe-bead-7ba6ecb45084-kube-api-access-t5gkp\") pod \"must-gather-f9z84\" (UID: \"baa0f01d-a3e5-4bfe-bead-7ba6ecb45084\") " pod="openshift-must-gather-ll6bw/must-gather-f9z84" Jul 02 19:49:16.935846 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:16.935804 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ll6bw/must-gather-f9z84" Jul 02 19:49:17.072908 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:17.072818 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ll6bw/must-gather-f9z84"] Jul 02 19:49:17.075574 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:49:17.075548 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaa0f01d_a3e5_4bfe_bead_7ba6ecb45084.slice/crio-63166cb7f468ab0ca85bbd9e20ababf44ff184fb631036d94fa4e5cbb8e05efd WatchSource:0}: Error finding container 63166cb7f468ab0ca85bbd9e20ababf44ff184fb631036d94fa4e5cbb8e05efd: Status 404 returned error can't find the container with id 63166cb7f468ab0ca85bbd9e20ababf44ff184fb631036d94fa4e5cbb8e05efd Jul 02 19:49:17.351779 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:17.351691 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ll6bw/must-gather-f9z84" event={"ID":"baa0f01d-a3e5-4bfe-bead-7ba6ecb45084","Type":"ContainerStarted","Data":"63166cb7f468ab0ca85bbd9e20ababf44ff184fb631036d94fa4e5cbb8e05efd"} Jul 02 19:49:18.361640 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:18.361598 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ll6bw/must-gather-f9z84" event={"ID":"baa0f01d-a3e5-4bfe-bead-7ba6ecb45084","Type":"ContainerStarted","Data":"669538d601c0d361a2dac9b2f62bc4140971343dfce92f6843b4e6ffda5ccb4a"} Jul 02 19:49:18.362133 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:18.361652 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ll6bw/must-gather-f9z84" event={"ID":"baa0f01d-a3e5-4bfe-bead-7ba6ecb45084","Type":"ContainerStarted","Data":"2299e1ac5141b29cbf129a8d278397810db5080b812f2bc54a625eb287022264"} Jul 02 19:49:18.382218 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:18.382151 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ll6bw/must-gather-f9z84" podStartSLOduration=1.6328185419999999 podStartE2EDuration="2.382131307s" totalImagesPullingTime="749.312765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:49:18.379977871 +0000 UTC m=+2497.495329095" watchObservedRunningTime="2026-07-02 19:49:18.382131307 +0000 UTC m=+2497.497482531" Jul 02 19:49:19.530732 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:19.530702 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jfl78_f714a87f-b1c3-4e57-8a2f-c1f61ab315be/global-pull-secret-syncer/0.log" Jul 02 19:49:19.715860 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:19.715823 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x6shn_315e9e92-0d3b-48f2-87d5-e3819e54fc0f/konnectivity-agent/0.log" Jul 02 19:49:19.796469 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:19.796395 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-165.ec2.internal_7a9b387f1ea494ba8e3440d0784de7c5/haproxy/0.log" Jul 02 19:49:24.862238 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:24.862183 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-s5gfq_2a9bc9fe-c25f-40af-b54c-b9ff2d91b22b/limitador/0.log" Jul 02 19:49:27.104764 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:27.104717 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slqtg_9e57abb6-b9be-42c1-9b2c-13d85e7700d8/node-exporter/0.log" Jul 02 19:49:27.126927 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:27.126891 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slqtg_9e57abb6-b9be-42c1-9b2c-13d85e7700d8/kube-rbac-proxy/0.log" Jul 02 19:49:27.154211 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:27.154169 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slqtg_9e57abb6-b9be-42c1-9b2c-13d85e7700d8/init-textfile/0.log" Jul 02 19:49:29.716231 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.716182 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx"] Jul 02 19:49:29.722481 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.722454 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" Jul 02 19:49:29.726972 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.726936 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx"] Jul 02 19:49:29.764773 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.764740 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/1.log" Jul 02 19:49:29.771368 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.771312 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-5pcbl_39433d94-da63-4417-8cac-bfe000cc31dd/console-operator/2.log" Jul 02 19:49:29.804053 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.804013 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b-sys\") pod \"perf-node-gather-daemonset-nwqrx\" (UID: \"449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" Jul 02 19:49:29.804331 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.804311 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b-proc\") pod \"perf-node-gather-daemonset-nwqrx\" (UID: \"449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" Jul 02 19:49:29.804618 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.804580 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b-lib-modules\") pod \"perf-node-gather-daemonset-nwqrx\" (UID: \"449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" Jul 02 19:49:29.804715 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.804644 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b-podres\") pod \"perf-node-gather-daemonset-nwqrx\" (UID: \"449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" Jul 02 19:49:29.804715 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.804685 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btw2w\" (UniqueName: \"kubernetes.io/projected/449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b-kube-api-access-btw2w\") pod \"perf-node-gather-daemonset-nwqrx\" (UID: \"449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" Jul 02 19:49:29.905871 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.905831 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b-proc\") pod \"perf-node-gather-daemonset-nwqrx\" (UID: \"449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" Jul 02 19:49:29.906192 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.906169 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b-lib-modules\") pod \"perf-node-gather-daemonset-nwqrx\" (UID: \"449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" Jul 02 19:49:29.906462 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.906330 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b-podres\") pod \"perf-node-gather-daemonset-nwqrx\" (UID: \"449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" Jul 02 19:49:29.906617 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.906598 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btw2w\" (UniqueName: \"kubernetes.io/projected/449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b-kube-api-access-btw2w\") pod \"perf-node-gather-daemonset-nwqrx\" (UID: \"449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" Jul 02 19:49:29.906772 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.906756 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b-sys\") pod \"perf-node-gather-daemonset-nwqrx\" (UID: \"449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" Jul 02 19:49:29.907281 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.906468 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b-podres\") pod \"perf-node-gather-daemonset-nwqrx\" (UID: \"449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" Jul 02 19:49:29.907432 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.906996 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b-sys\") pod \"perf-node-gather-daemonset-nwqrx\" (UID: \"449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" Jul 02 19:49:29.907543 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.906031 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b-proc\") pod \"perf-node-gather-daemonset-nwqrx\" (UID: \"449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" Jul 02 19:49:29.907637 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.906441 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b-lib-modules\") pod \"perf-node-gather-daemonset-nwqrx\" (UID: \"449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" Jul 02 19:49:29.915896 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:29.915859 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btw2w\" (UniqueName: \"kubernetes.io/projected/449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b-kube-api-access-btw2w\") pod \"perf-node-gather-daemonset-nwqrx\" (UID: \"449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b\") " pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" Jul 02 19:49:30.036501 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:30.036381 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" Jul 02 19:49:30.228996 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:30.228948 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx"] Jul 02 19:49:30.233270 ip-10-0-142-165 kubenswrapper[2565]: W0702 19:49:30.233222 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod449aaa56_3ce3_4cd7_b0c6_dd13b1d92e9b.slice/crio-7e597a75b2e529f98e7b0931415d166e339f730a3e3b4169a421892c3c78cea7 WatchSource:0}: Error finding container 7e597a75b2e529f98e7b0931415d166e339f730a3e3b4169a421892c3c78cea7: Status 404 returned error can't find the container with id 7e597a75b2e529f98e7b0931415d166e339f730a3e3b4169a421892c3c78cea7 Jul 02 19:49:30.465770 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:30.465723 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" event={"ID":"449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b","Type":"ContainerStarted","Data":"bfb8f3160c5cf9ec6a501018ab0da9292254fffd23524c3d48c109c06dbf42e0"} Jul 02 19:49:30.466129 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:30.465833 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" event={"ID":"449aaa56-3ce3-4cd7-b0c6-dd13b1d92e9b","Type":"ContainerStarted","Data":"7e597a75b2e529f98e7b0931415d166e339f730a3e3b4169a421892c3c78cea7"} Jul 02 19:49:30.466129 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:30.465894 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" Jul 02 19:49:30.485424 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:30.485366 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" podStartSLOduration=1.485330505 podStartE2EDuration="1.485330505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:49:30.483656656 +0000 UTC m=+2509.599007901" watchObservedRunningTime="2026-07-02 19:49:30.485330505 +0000 UTC m=+2509.600681735" Jul 02 19:49:30.993689 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:30.993655 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-gqj2t_54667ba4-1e72-488c-9fe3-53026b8d2df9/volume-data-source-validator/0.log" Jul 02 19:49:32.089541 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:32.089514 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8fcgj_442ba05b-cbb8-48fb-b1ab-96ebe4960f80/dns/0.log" Jul 02 19:49:32.112395 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:32.112316 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8fcgj_442ba05b-cbb8-48fb-b1ab-96ebe4960f80/kube-rbac-proxy/0.log" Jul 02 19:49:32.206428 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:32.206401 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dqx74_e9bceedd-6e94-4988-baf5-5eb9eb417a7b/dns-node-resolver/0.log" Jul 02 19:49:32.773372 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:32.773302 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-544bbd9b4c-lw6hf_043ef48e-d5b2-4090-af93-39a4a503f8d1/registry/0.log" Jul 02 19:49:32.828070 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:32.828019 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hf8w8_b3b59a24-8f2e-4753-a3b2-fb43e2e6cd39/node-ca/0.log" Jul 02 19:49:34.082011 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:34.081915 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bdf57ddb6-5k2qt_ef604999-a8e7-41ac-88ef-c5465fe39c9d/kube-auth-proxy/0.log" Jul 02 19:49:35.028033 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:35.027996 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wlnv6_e81d42fb-8cb6-4e07-863d-4912aa0ed5e8/serve-healthcheck-canary/0.log" Jul 02 19:49:35.618710 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:35.618673 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2tmvt_4996d3a4-2f90-403e-a277-df9983b8e7f0/kube-rbac-proxy/0.log" Jul 02 19:49:35.644765 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:35.644740 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2tmvt_4996d3a4-2f90-403e-a277-df9983b8e7f0/exporter/0.log" Jul 02 19:49:35.666300 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:35.666270 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2tmvt_4996d3a4-2f90-403e-a277-df9983b8e7f0/extractor/0.log" Jul 02 19:49:36.484529 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:36.484495 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ll6bw/perf-node-gather-daemonset-nwqrx" Jul 02 19:49:38.429824 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:38.429770 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d8ddf44b5-cw5hx_d7ae933f-25af-4598-8fd8-4cb71a8ac1d1/manager/0.log" Jul 02 19:49:38.450862 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:38.450824 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d8ddf44b5-cw5hx_d7ae933f-25af-4598-8fd8-4cb71a8ac1d1/copy-manifests/0.log" Jul 02 19:49:40.200376 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:40.200272 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7d75bb4b7f-62xj5_a1a4e1ea-db6b-4c46-a318-090d19359a2f/manager/0.log" Jul 02 19:49:40.229046 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:40.229007 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-bwvhz_9536d1c3-68cd-4471-80dd-087cfb7b7391/openshift-lws-operator/0.log" Jul 02 19:49:40.651112 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:40.651080 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bdf57ddb6-5k2qt_ef604999-a8e7-41ac-88ef-c5465fe39c9d/kube-auth-proxy/0.log" Jul 02 19:49:42.310987 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:42.310955 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bdf57ddb6-5k2qt_ef604999-a8e7-41ac-88ef-c5465fe39c9d/kube-auth-proxy/0.log" Jul 02 19:49:45.736241 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:45.736192 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-gnf78_da2642bb-e400-40ec-bded-5f145bbba018/kube-storage-version-migrator-operator/1.log" Jul 02 19:49:45.736987 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:45.736965 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-gnf78_da2642bb-e400-40ec-bded-5f145bbba018/kube-storage-version-migrator-operator/0.log" Jul 02 19:49:46.821855 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:46.821782 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgwb4_ce6ec4d4-1f88-40e4-b020-e1e137efd53a/kube-multus-additional-cni-plugins/0.log" Jul 02 19:49:46.843157 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:46.843128 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgwb4_ce6ec4d4-1f88-40e4-b020-e1e137efd53a/egress-router-binary-copy/0.log" Jul 02 19:49:46.864720 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:46.864694 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgwb4_ce6ec4d4-1f88-40e4-b020-e1e137efd53a/cni-plugins/0.log" Jul 02 19:49:46.891060 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:46.891028 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgwb4_ce6ec4d4-1f88-40e4-b020-e1e137efd53a/bond-cni-plugin/0.log" Jul 02 19:49:46.919845 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:46.919814 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgwb4_ce6ec4d4-1f88-40e4-b020-e1e137efd53a/routeoverride-cni/0.log" Jul 02 19:49:46.947194 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:46.947159 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgwb4_ce6ec4d4-1f88-40e4-b020-e1e137efd53a/whereabouts-cni-bincopy/0.log" Jul 02 19:49:46.986385 ip-10-0-142-165 kubenswrapper[2565]: I0702 19:49:46.986264 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dgwb4_ce6ec4d4-1f88-40e4-b020-e1e137efd53a/whereabouts-cni/0.log"